Making the Cloud, Microservices, and Enterprise IT work!

Data Gravity and Cloud Uplift Woes

2021-03-25T10:30:00.000+02:00

When I originally learned about Data Gravity via David Linthicum’s excellent podcasts, a key architecture point that stuck in my mind was: your application needs to be close to its (interactive) data sources. (Data Gravity has since picked up a wider yet less useful definition as more applications cluster together in a “data galaxy” to be close enough for large amounts of data to be able to interact.)

Why? Every interaction with a database has communication (network time) overhead. Most applications are built with their servers and database on the same LAN / subnet / vnet – usually in close physical proximity, specifically so that time is minimized. Every hub/switch/router adds time to the request, so while a request may pay 10ms when it’s local, add time per “hop”.

Application performance and tolerances are implicitly built around that response overhead. If data takes too long to return, the developers will likely adjust their code to do bigger queries, wider joins, etc. But what’s “too long”? If a user is waiting for an application response, “too long” is whatever the user tolerance is. If it’s a background process, the tolerance is likely much higher as long as the process can complete within its time window the overhead may not even be noticed.

In the early discussions of Data Gravity, the questions were often around “can I move my expensive database cluster to the Public Cloud while leaving my application on premises?” And the answer was “no – it would add too much communication overhead, damaging the applications’ response time.”

Years later we’re in a different position. Enterprises are mass uplifting applications. Some are up in the cloud, some left behind in the data center.

And suddenly many applications are encountering data latency.

When applications are interacting by direct database access to other applications, particularly interactive interactions (many queries as opposed to bulk queries), essentially they are tightly coupled. And if one is in the cloud and another left behind in the data center, they’re subject to all the network overhead as well as any bandwidth limitations that may exist between the data center and cloud. (Azure mitigates the bandwidth issue somewhat with “Express Route”, AWS with “Direct Connect.)

The bad news is that if the application has a frequent interaction pattern with the remote databases, there may be no good solution other than refactoring the application (and it’s often mature applications that may be in maintenance mode that are being uplifted – meaning it’s a problem) – particularly if the latency is causing unacceptable user impact. If the problem is bandwidth, it will be time to invoke the network team to assure the traffic is following the optimum path, getting prioritized, and that sufficient bandwidth is available to meet the demand.

The summary is, before uplifting an application that uses direct database access to other applications/databases (whether code based, remote SQL, or something like SSIS), OR if the application is accessed via direct database access by other applications, check the interaction pattern to determine if additional latency is going to unacceptably impact performance.

[ There's another whole solution direction I'm avoiding that only applies to large enterprises, namely AWS Outposts and Azure Stack, bringing the edge of the Public Cloud into physical proximity with the enterprise data center by placing a rack of AWS/Azure equipment into the data center. With close local routing, both bandwidth and latency issues are avoided. At the time I'm writing this, AWS is creating mini-outposts, which may make it a viable option for smaller enterprises. HOWEVER, this also assumes that organizations are going to be comfortable directly connecting AWS or Azure managed equipment (and networked back to them) into their data center. I can see the cyber-security team heads exploding from here. ]

Serverless - Unintentional Granularity Problems

2021-01-06T20:26:00.003+02:00

Serverless is great, but serverless functions rarely run by themselves... they're usually connected to data sources - and frequently that means SQL databases.

In the graphic attached, my cloud SQL database is collapsing under assault by a serverless function that's scaling up instances to meet demand. Here's how we unintentionally created this problem:

To allow for parallel processing to speed processing, we've taken a transactional billing file, broken it into groups of 30 transactions, loaded those into messages and queued them. We then have a serverless function that's listening for these events, and if messages are still waiting after a certain amount of time, another instance is spawned to begin handling the waiting messages and processing.

When it was sent thousands of transactions, this worked great and the business users were suitably impressed with the processing speed (which dropped from many hours to a few minutes). When they began sending larger transaction sets, tens of thousands, the processing speed remained great but the single instance database CPU began spiking. No problem - it's cloud so size up the database instance (though don't ignore the very significant cost jump of moving from small instances to medium or large ones).

But when it was sent hundreds of thousands of transactions, the single instance database hit capacity. [ This is a common traditional relational database problem -- you hit capacity of CPU, Memory or IOPS - even if you size up your database you can only push so much volume through it. This can be stretched with sizing up CPU and Memory if it's complicated or non-optimized SQL's, and with cloud databasing provisioning more IOPS if it's poorly indexed or doing huge reads and writes, but quickly gets very expensive and still has it's 'not too big' limits. There are some tricks such as creating read replicas that can extend the ability to use a traditional SQL DB model. But that's not the focus of this article.]

Serverless functions are "throttle"-able (in AWS this is called concurrency), and concurrency is set-able per serverless function, and this is the traditional way to handle this sort of problem. (This limits the performance, but there's always trade off's). But this comes to the point of this article...

Concurrency is AT THE LEVEL of the (serverless) function. If, for example, your function contains the code that handles many different processing situations or transaction types, because they are combined in the single deployed function, they can only be throttled at the level of the combined functionality. (If you wanted to allow up to 100 instances for transaction type one, but limit to 10 for a less time sensitive transaction type two, you can't segment it because the control is at the overall deployed function.)

Meaning part of the consideration for how to break down and deploy serverless functions needs to include at what level of granularity is needed to apply the levers of control (settings such as concurrency and function resource sizing) and monitoring.

In the graphic above, the team is using a serverless application framework (Laravel Vapor) that deploys the full application into 3 separate functions: one responds to user interaction, one to messages, and one to command line interaction. So it is impossible to throttle without providing the same limit, a combined limit, across a processing mode. This team has some refactoring to do.

The granularity of the function deployment affects your ability to resource manage (and therefore the cost), as well as control and monitor of your functions.

Architect and developer beware.

Cloud Native is already Yesterday

2019-05-02T14:31:00.004+03:00

Cloud hosting, de-coupling (server) virtualization from the supporting hardware and moving it to a Public Cloud (letting the physical tier and network tier be someone else’s problem) is old news. Plenty of IT shops are still struggling with what to move, when or if to shut down their data centers, and the issues of data gravity… regardless, Cloud hosting has passed mainstream adoption and moved far into the Late Majority.

The surprise is, Cloud Native, replacing ‘virtual machines’ and the implementation of application server (software), database server (software) and the like with “As A Service” Public Cloud services is also far along. And regardless of how far in the adoption cycle it is, it is far along in technology maturity. Far enough that the Public Cloud vendors have already built the next generation upon it.

The next generation and next cycle, Cloud as the Application Service Layer, is already here, already market viable, and already becoming the base for the next generation of applications. And leveraging Cloud Native and Cloud Application Services, the pace of the change has significantly accelerated.

Technology & software changing at pace is nothing new. Every mid to large IT enterprise struggles with a tech portfolio and its associated life cycle. And almost no organization can afford to refresh its full tech portfolio and ends up managing generations of software and technology. The shift to Cloud is not only a technology shift, it’s also creating increased pace of the change cycle.

The shift to Cloud started straightforward… once virtualization became the data center tech stack, the deployment of virtual machines and their resources (memory, disk, cpu) was already disassociated from the physical platforms. There were (and still are in many a private data center) system and resource admins to manage and plan the resource allocations and supporting equipment purchases, and deployments, and configurations, and patching, and maintenance, and and and…

The move to the Public Cloud moved the management from those system and resource admins to the Public Cloud provider. The management and maintenance of those resources, the planning, ordering, and installation, patching, and and and… became someone else’s problem. The IT organization could focus on the control and cost, but no longer has to be concerned with the management and operation of the underlying equipment, network, infrastructure, etc.

Which is awesome. Jump to the Public Cloud, effectively outsource control of the lowest level of IT infrastructure – and retire the majority of the physical data center… let Amazon or Microsoft or Google build and maintain it. For IT enterprises, that’s the initial migration to the Cloud. (This was initially referred to as IaaS, Infrastructure as a Service, when vendors and architects were trying to categorize cloud capabilities.)

And then came Cloud Native.

The Public Cloud vendors began providing database services, queue services, storage services and more. If an IT organization needs a database, it could have a system admin spin up a virtual machine, a database admin install the database software, set up backups, cluster it, set up monitoring, and it’s ready to be used… then bring in a DBA to tune, optimize, periodically patch, yearly version upgrade – cost cost time cost, all overhead (no direct business value).

Or use a Cloud “database as a service”, pick the type of database needed and have a Cloud database at the ready. The Cloud provider spins up the capacity, the database instance, offers a selection backup options, clustering options, and keeps it patched and version upgraded.

Take the underlying infrastructure components and use Cloud “as a service” offerings, and be Cloud Native. (Is this advanced IaaS or moved up to Platform as a Service, the formal industry answer is “yes”.)

But now the Cloud Providers have moved both up and down the stack – offering more Cloud Native capabilities and offering Cloud as the Application Service Layerbuilt on their own Cloud Native stack.

Down the stack:

Build, install and manage a container manager (aka Docker and/or Kubernetes), or use one of various “Containers and Kubernetes as a Service” options(for example AWS Fargate which auto-manages most aspects of the container environment – at an additional cost, but significantly reducing the need for expertise and operations staff). This effectively lets a team use “containers” as if they were serverless (but allows for long running processes).

Serverless takes this to a greater extend, allowing teams to deploy and run code without any infrastructure setup – virtual or otherwise. (But has limitations on running time, making a place for both containers and serverless depending on the business and programming approach used to meet it.)

Up the stack:

Code and security validate an application security structure (with supporting LDAP database) or use “User Management as a Service”(such as AWS Cognito). Instead of weeks to months to build user management (and it would be wise to have it security tested), have a fully capable service added that is security certified completed in days to weeks.

On top of the stack:

Take services being offered, such as AI services (built using cloud native database services built using basic cloud virtualization services), build up the model and offer business capabilities…as a service. Examples such as AWS Comprehend (OCR interpretation of scanned documents) or AWS Comprehend Medical (OCR specifically focused on medical documents), or AWS Lex (voice to text – voice enable an application).

Cloud as an Application Service Layer means moving from building with legos to building with lego kits.

What’s the impact to the IT enterprise?

- Much faster application development, with supporting capabilities being used (as a service) rather than developed. This is a boon to the business, but will pose challenges for IT to adjust processes to accommodate the pace (CI/CD, continuous and automated QA, and user training).

-      Decrease in operations costs (though a significant portion of that will transfer to the bill for the cloud vendor services) through eliminating equipment, data centers, and lower level operations personnel.  This may mean repurposing staff, retiring vendor relationships, and figuring out how to manage software licensing in a dynamic environment (with each software vendor having different approaches).

-      A need to build up management capabilities around the services being used.  While cloud resources are easy to activate, they are tricky to track and de-activate.  Bills can grow and unused resources not be released if it’s not tracked and managed.

-      Change change and more change.  While IT has been trying to manage operations and security on the existing tech portfolio, the cloud vendors have pretty much had an open field with no legacy portfolio to support.  They’ve applied their full capabilities to creating new features, and are bootstrapping – showing how fast tech can move based on Cloud Native.  AWS Comprehend Medical shows an example of veering directly into business features.

The use of Public Cloud is only going to accelerate, and their Application Server Layer capabilities plus AI and Machine Learning capabilities are quickly moving software into new capabilities. Is your organization Cloud Native+ ready?

Enterprise Strength Integration - as of 2011

2018-08-13T22:34:00.001+03:00

Time for another historical integration presentation.

From 2011 to a major corporate client, visually less pleasing (and I had to reformat it off the customer’s corporate template), but the information value is high.

Every presentation is contextual, but I think this one's got some pretty strong hints.

Ongoing credit to @Hillel Fuld for inspiring me to contribute, and congrats on his new https://www.hillelfuld.com. While I've seen the value of sharing and contributing in private live and community, Hillel demonstrates the strong value of sharing in the business world and being a mench.

While these presentations are dated, I think they offer value understanding where the tech world was, how it has built up to where it is, and some key ideas that can make a difference in projects today.

Enterprise Strength Integration (as of 2011) from Akiva Marks

No SOA ROI - SOA is Dead? Getting real ROI from Integration - as of 2009

2018-08-12T21:05:00.001+03:00

Continuing sharing of my historical enterprise architecture presentations, with a strong focus on enterprise systems integration.

Ongoing credit to @Hillel Fuld for inspiring me to contribute.

While dated I hope they offer value understanding where the tech world was and how it has built up to where it is.

This presentation is older, 2009, less flashy and more wordy. But may actually take you deeper through enterprise integration concepts. Please comment if there are any points you'd like to discuss.

No SOA ROI - SOA is Dead? Getting SOA Value from Akiva Marks

SOA Methodology & Strategy - as of 2010

2018-08-10T17:24:00.002+03:00

I've decided to share my library of presentations focused across my years in enterprise architecture in a US Fortune 50 corporation and my years as an enterprise architecture & integration consultant and IT management consultant in Israel. I hope you enjoy.

While they may be dated they may be of value to understand where some of our technologies and software catalogs of today are at, and where there can be some improvement.

Here's the first... SOA (Service Oriented Architecture) Methodology circa 2010.
I credit Hillel Fuld's recent interview for the inspiration to share.

SOA Methodology - Strategy (as of 2010) from Akiva Marks

Amazon S3 Easy Scripted Backup from Windows for the Enterprise

2016-07-11T17:14:00.001+03:00

I don’t normally post code, nor am I normally implementing scripts myself. But sometimes to learn the ins and outs of a capability, you have to dive in and try it out. Since I’ve been working with Amazon Web Services (AWS) via Windows, I’ve found a remarkable lack of sample scripts for using it, so I’m posing my little project here.

For me, with heavy Unix scripting experience in my distant background, using Powershell and the AWS Powershell add-in was a no brainer. While the syntax of Powershell is significantly different from Unix Bourne shell, the capabilities are practically identical, including piping.

Now for the requirements:

- I needed to backup to the Cloud for an offsite backup.
- The data needed to be encrypted with a client-managed key, but I had neither the tools nor onsite CPU or extra storage for client-site encrypting.
- The backed-up data needed to track changes or access.
- To show any modifications, the backed-up files needed to manage versions – so no changed version would overwrite a previous version.

Here’s what I did…

1. Set up a specific IAM user with permissions only for S3 by:

- Created an AWS group titled “backup_group”.
- Attached the policy “AmazonS3FullAccess” and no others.
- Created users “backup_user1” & “backup_user2”.
- Stored these user’s REST access keys in a secure encrypted local location.
- Added both users to “backup_group”.

2. Create a specific S3 bucket(s) for these backups, accessible only by the Backup user (and administrative user), with logging.

- Created S3 bucket backup-logs with lifecycle setting, DELETE all content 2 years old or older – meaning logs have a 2 year life. (If you don’t give log directories a lifecycle rule, they’ll accumulate forever with ever increasing storage costs. Since this “disk” never “fills”, you won’t get any kind of log error that would force rotation, just an ever growing cost.)

- Created S3 bucket “backup-for-bi”, enabled logging to “backup-logs” with subdirectory logs-bi/
- Created S3 bucket “backup-for-DB”, enabled logging to “backup-logs” to subdirectory logs-db/

3. Enable versioning to preserve each copy and prevent hidden changes. – enabled on both buckets.

6. Utilize an upload command setting that requires encryption of the uploaded data with an client managed key, which will prevent any unencrypted download of the content even by Amazon. Key will be stored locally.

- Now this was particularly tricky and confusing, encryption keys not being my specialty. AWS’s data encryption is AES 256, but generating an AES 256 key was rejected. The command spec says it should be Base64 encoded, which I did but it was still rejected. In the end I was able to generate an AES-128-cbc encrypt key from a passcode, and then Base64 encode that key which generated a 44 byte string (ending with =) that AWS accepted. In essence, that Base64 string is, as far as we are concerned, the key – though I’m storing that key, the original password, and the 128 bit key and salt.

With all that prep ready, here’s the Powershell to upload a list of directories, the list being embedded in the shell. $accesskey equals the AWS IAM user access key shown by AWS on creation of the user. $secretkey is also shown by AWS on creation of the user.

<#
.DESCRIPTION
    Upload Listed Directories to Amazon AWS S3
    
.NOTES
    PREREQUISITES:
    1) AWS Tools for PowerShell from http://console.aws.amazon.com/powershell/

.EXAMPLE
    powershell.exe .\AWS_Backup_Dirs.ps1  
#>

$bucket          = "nameofbackupbucket"
$backup_list     = "E:\Prod", "E:\PreProd"
$AES256_key      = "AAAABBBBCCCCDDDDEEEE99991111222233334444777="
$accesskey       = "ASDKLJASDFJKLASDF"
$secretkey       = "ASDLFJKWEIOPUQWERASDFJ/LASDFASDFASDFASDF"

try
{
    import-module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"
}
catch [system.exception] 
{
    $error_fail = "Error: AWS Powershell Extensions not installed or missing from expected location... " + $_.Exception.Message
    Write-Host $error_fail
    throw $error_fail
}

foreach ($backme in $backup_list)
{
    $bucket_subdir = Split-Path -Path $backme -Leaf
    try
    {
        Write-S3Object -BucketName $bucket -Folder $backme -Recurse -KeyPrefix $bucket_subdir -AccessKey $accesskey -SecretKey $secretkey -ServerSideEncryptionCustomerProvidedKey $AES256_key -ServerSideEncryptionCustomerMethod AES256
    }
    catch [system.exception] 
    {
        Write-Host  "Error: " $_.Exception.Message
    }
}

Notes:

The –KeyPrefix specifies that the data will be written into subdirectories matching the last name of the directory path. So if the path is D:\dog\cat, it will be stored on S3 in the subdir “cat”.

This script can be set up as a scheduled task and run daily or weekly. BUT, as written it will send up the whole directory, which can bear the cost of the full data transfer even if nothing has changed. If you want incremental backups, you have to adjust the script to only find newer files, and loop to send them up one at a time (rather than the whole directory in this script).

Hope this helps!

Cloud Flexibility encounters IT Procurement Inflexibility

2016-06-06T17:24:00.001+03:00

The Cloud. Whether it’s a mega-cloud provider in the public clouds, a private or managed cloud, on premises or off prem, cloud is all about flexibility. Add an instance, add a service, it’s just a click.

(Note the dynamic cloud, the ability of an app to dynamically expand it’s resources as load increases, remains mostly hype. While this was one of the first great promises of the cloud-o-sphere, it has not translated into reality. Even further, just shutting down non-production environments during the night or on a schedule can be a significant cost saver – but is not offered by the mega-providers. At least in this area 3rd party cloud support vendors have stepped in – but many are not aware of this and end up with lots of idle time on compute nodes.)

While the cloud providers offer that wonderful relatively instant service with a click, each one of those clicks carry a cost. And cost means…IT procurement, the department focused on getting the best deal for their IT dollar and making those long term contracts that keep us operational. And they have a process, a long process, for each item to be purchased.

Cloud flexibility means I can just add a node or VM, and add a backup or DB or firewall. IT procurement means forms and weeks and reviews.

When we were purchasing compute capacity for the project for the year, which consisted of a series of servers and expensive software licenses, this made sense. My purchase had significant cost and long term implications.

With Cloud accounts, my “purchase” can be “unpurchased” at any time (at least in the public clouds – private clouds often require some time commitment), and it can start small and grow as the capacity need grows. (In traditional IT, “new projects” often purchased their first 2 years of servers in their initial purchase – that’s how it was done, nobody wanted their project to be in trouble due to insufficient resources, which often meant over purchasing, and until the project was in production under actual production load they were often unclear of the actual server capacity that would be needed beyond a high level guesstimate…that could easily be 50% off.) With Cloud, we can start small and increase capacity with ease as the actual usage grows.

Obviously a significant positive NOT to have to buy capacity we’re not using today and may not use for 18 months. AND we can increase to the ACTUAL need rather than a predicted need 2 years ago. BUT…if we have to go through a full IT procurement process as we make those Cloud changes, we’re hobbled and unable to gain that value.

This is not theoretical. Try having a cloud vendor procedure a Purchase Order and Statement of Work for “Add Managed Backup to Node, $129.95 per month” and “Storage Encryption Service, $39.95 per month” – the experience isn’t pleasant for anyone and makes Cloud use somewhat impractical UNLESS we return to the old approach of over-estimate everything we need and buy (allocate all those high volume nodes and services) up front.

It’s easy to overuse, over allocate, and not manage the cloud resources (not release resources and services no longer in active use), and IT procurement can provide necessary oversight and management of cloud resources. But they have to come with new cloud flexible procedures to do so.

Otherwise, the value of cloud services is lost.

Continuous Integration vs. Micro-Services

2015-11-25T18:49:00.001+02:00

I was reading Mike Kavis’s Do This, Not That: 7 Ways to Think Different in the Cloud and encountered what initially sounds like reasonable advice…but becomes absolutely unrealistic. I’ll explain why at the end. Mike writes…

Think Empower, Not Control

The first thing many companies do when they start their cloud initiative is figure out how to lock it down. Too often, the people who own security and governance spend months (sometimes years) trying to figure out how to apply controls necessary to meet their security and regulatory requirements. Meanwhile, developers are not allowed to use the platform or worse yet, they whip out their credit card and build unsecured and ungoverned solutions in shadow clouds.

We need to shift our thinking from “how can we prevent developers from screwing up” to “how can we empower developers” by providing security and governance services that are inherited as cloud resources are consumed. To do this, we need to get out of our silos and work collaboratively. Instead of enforcing security and governance controls by requiring rigorous reviews, we need to bake policies and best practices into the SDLC.

Start with continuous integration (CI). Automate the build process and insert code scans that enforce coding best practices, security policies, and cloud architecture best practices. Fail the build if the code does not meet the appropriate policy requirements. Let the developers police themselves by using automation that relies on policies established by the security, governance and architecture teams. Set the policies and then get out of the way, letting the build process do the enforcement. Developers will get fast feedback from the CI process and quickly fix any compliance issues – they need to or the build will never get to production.

Once applications are deployed, run continuous monitoring tools that look for violations or vulnerabilities. Here’s a novel idea: Replace meetings with tools that provide real time feedback.

Ahh, the magic of the perfect Software Development Life Cycle empowered by the perfect DevOps continuous integration. That will surely solve the problems of breaking down silos and collaborative working, right?

Sadly I’ve yet to encounter a technology or methodology that magically restructures the IT organization. The IT organization structure has come into place due to business, IT management, and cultural drivers of the organization. There are technology changes that have forced restructuring, but it’s always painful and time consuming. Breaking silos is one of the hardest (as this usually challenges political control and authority structures – and people are loath to give up control / authority / influence.)

Mike focuses on the second part and skips a key idea he presents in the first part. Namely, give the developers the environment that’s the target. Or, expanding on this idea… ARCHITECT your systems for their target environment. Systems architected for cloud or hybrid scenarios (should it matter to the application if it’s deployed locally, locally on dedicated servers, VM’s, soft partitions, private cloud, public cloud…or a mix of all the above?)

Systems can easily be architected for distributed deployment…IF that thought goes into the requirements early on. This may mean messaging and/or event driven architecture instead of real-time web services, even layering internal components to pass events instead of locally calling or instantiating. By doing so the architecture becomes micro-service oriented, which at the macro level means component groups can be bundled into deployment packs and deployed across the various server / resource models as needed.

It’s not can we break down the silos between people and teams – which may be appropriate but long and painful and STILL result in unwieldy systems. It’s how we model the interaction between and within the silos that will give the flexibility to deploy anywhere and coordinate/communicate/integrate practically automatically.

Micro-Services vs. Business Service Granularity

2015-09-16T12:27:00.001+03:00

In the many discussions and architecture approaches I’m reading about Micro-Services, a key point seems to elude the conversation. Namely, “What’s a Micro-Service?” Not what are the technical properties of a micro-service, but rather what level of BUSINESS functionality should be encapsulated in a single service, a single “micro” service?

One “formal” definition that’s floating around is this…

...an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies. - James Lewis and Martin Fowler via http://techbeacon.com/

From the SOA (service oriented architecture) perspective, decomposing an application into it’s composite transactions and business objects, and wrapping them in accessible services, has been the trend for the past 15 years or so. Applications have been turning into business engines, and in some cases the interface layer itself has been successfully moved from having low level interaction with the business code to using transaction / object / service layers. Seeing an application or platform or suite that offers a full catalog of accessible transactions and objects (wrapped in services) has become the norm (and even at the code level, structuring the classes in a similar way).

Example – Oracle e-Business Suite service catalog…

The micro-services architecture concept comes along and says “no no no, services should stand on their own”, to the technical point of being divorced from their implementation platform and container. But at what level of granularity should BUSINESS SERVICES be implemented? Can we build “virtual” applications that are logical composites of many “business micro services”, or do business transactions remain grouped as the combined code we refer to as “applications” and/or rolled up into “suites”?

Years ago I was giving a presentation describing how applications are decomposing, as they become integrated with and real-time linked to other applications and thereby becoming distributed applications – with a follow up discussion of integration reliability, monitoring, control, security, etc. The primary change of that generation was the application no longer standing on it’s own, even though that “it’s own” may have been data being imported or updated from other systems regularly. The distributed application moved data out of it’s own system and onto an integration connection point, with the advantage of making the data (or transaction) much more up-to-date – (semi) real time data and transaction movement.

But, my customers asked me, what if we are development NEW applications? My answer - we would want to bundle the functionality differently, to allow more flexible combinations.

• Break our “Application” model into…

− Transaction Components

− Process Components

− Entities

− all exposed as Services

• Compose, Coordinate, Combine them into…

− Business Processes

− User Presentations

− meaning “Applications”

At the time, this still presented signification issues of managing, controlling, tracking, and securing. API Management and Micro Services would seem (in theory, it’s early days) to resolve these issues. Which means we can get to a direct discussion of granularity of BUSINESS SERVICES, allowing for BUSINESS ORIENTED MICRO SERVICES.

A Business Unit has one or many Business Capabilities. A Business Capability is a series of one or many Business Processes. And a Business Process has one or many steps, each being a Business Service. From the IT perspective…

The Businesses processes map to Workflows or Business Process Automation’s from the IT implementation perspective. In older applications workflow was encoded in the application itself. Nowadays if the Business Services are encapsulated and exposed, they can be orchestrated externally via BPM (business process management), BPA (business process automation) or other integration layer controllers.

The question is, of what granularity should those Business Services be, and therefore the corresponding IT Services? How “micro” should a “micro-service” be? Is “Invoice Management” with “Create Invoice / Query Invoice / Update Invoice / Pay Invoice” a (large) micro-service, or is “Invoice Object” a micro-service encapsulating only the persistence and structure of the invoice, and Create / Query / Update / Pay Invoice each being a separate micro-service? (With the connectivity overhead and external orchestration overhead). And would breaking the functionality down to that level of granularity offer a sufficient value to compensate for the added overhead?

The answer is unclear. The pure technology proponents would tell us to go full granularity, and not concern ourselves with the connectivity overhead – as that’s a container and implementation level problem – nor concern ourselves with the orchestration overhead, as that’s an analysis and mapping problem (with it’s own implementation overhead). The deploying, hosting, and tracking of all those services should also be “easily” handled and managed by the environment. We’ll see.

Martin Heller at TechBeacon writes, “If a service looks cohesive and deals only with a single concern, then it's probably small enough. If, on the other hand, you look at a service interface and see a number of different concerns being combined, perhaps it's a candidate for further decomposition. At the other end of the spectrum, if the service doesn't do enough to feel useful, perhaps you overdid the decomposition and need to combine it with a related service. It's very much like the game of "find the objects" that people played when designing object-oriented software 25 years ago, but now the objects are services, not classes.”

I believe this ties it nicely with the picture above. A “business service”, such as Invoice Management, with it’s functions of create / query / update / pay, composes nicely into an IT service…and therefore (in the new terminology) a “micro-service”. The only reason to decompose further would be IF there was a desire or ability to substitute the granular functionality from other developers / development teams / or application vendors. And while that’s conceptually feasible, managing service catalogs at that level is simply not (yet) reasonably viable.

What this says to me is that the ONLY difference between Micro-Services and the SOA Services of last year is the expectation that the corresponding code is sufficiently encapsulated to be independently deployable. And while that independent deployability offers some interesting theoretical advantages, those advantages bring major management and control issues that have not been doing so well in practice. Example – managing the service catalog. SOA Governance, particularly design time governance, has not been successful in the field. Few IT enterprises have gotten the ROI from the vendor offerings in this space. API Management seems to be changing that, but still the point is managing and coordinating hundreds to thousands of coordinating services is a daunting task.

I would advise stepping carefully into Micro-Services. Try a few small projects to understand the dynamics of the use pattern AND the management pattern. There are advantages to be had, but risks as well. It’s likely we’re seeing the future of software development. But the control structures and supporting architecture patterns are not yet in place. Tread carefully.

Big Ball of Mud Software

2014-12-31T14:51:00.001+02:00

In the space of Software Architecture, the “Big Ball of Mud” represents “natural growth” – or the system that just adds and changes without ANY planned architecture. (More on the Big Ball of Mud here.) While we hear about it, and sometimes run into it as we have to solve project problems, how do you spot a software product in that mode?

Side note… while traditionally a Big Ball of Mud is discussing gradual changes to a system or program, we also see a Big Ball of Mud in enterprise architecture in unplanned natural growth of the addition of various systems and technologies, and interfaces and interconnections between them. While dealing with spaghetti code is tough, dealing with spaghetti connections and systems is extremely expensive and risky – but is all too frequent.

Here’s a software product conversation I had this week…

Please wait for a site operator to respond. You are now chatting with 'Randy'. Your Issue ID for this chat is LTK1219208815693X

Randy: Welcome to Unnamed Product Corporation Sales Support. My name is Randy, how may I help you today?

Akiva: Hello Randy. I wish to sync Outlook between my Outlook, which is on my office network and exchange service, and an Outlook that's on an isolated (no internet connection) secure network. So my questions are:

1 – is Product the tool for this (I actually only need to sync calendar)?

2 - will I be able to install it without an internet connection (since the sync computer is on a non-internet-connected network)?

(isolated computer does have a normal USB)

Randy: Product has a feature to synchronize Outlook with a USB stick. But to install Product Internet connection is necessary.

Akiva: would seem to reduce the value of a USB sync. (In other words, WHY IN THE WORLD would you sync by USB stick if you have an Internet connection?)

Randy: Could you please clarify your two computers are connected to the same network?

Akiva: no they are not. Computer 1 is a normal internet connected laptop, connecting to an exchange server in office 1. Computer 2 is on a disconnected private network with a private exchange server, with no external network connections.

Randy: In this case the only option to synchronize your Outlooks is USB stick profile (great, this is what I want to do), but to install Product is necessary Internet connection.

Akiva: Can you please confirm that? Your download page as a link that says "Full Standalone Installation* *Internet connection is not required during installation"

Randy: Just a second. I just talked with our technician, he confirmed that you are able to install Product without an Internet connection, sorry for the mistake.

Akiva: It's ok. However, I’m trying it – it says it requires a particular KB patch to .Net 4.0 as a supporting component - which it tries to download. and of course fails - the computer has no internet. I'm now trying to download and pre-install that component. (So tell your tech you were 1/2 right.)

Randy: You are able to download Product trial version which has 14 days trial period and try it.

Akiva: Ok, important question for the tech. When I buy it, does it require an internet connection to verify/certify/check the license id you provide?

Randy: Internet connection is required to activate your license. Also to renew it.

Akiva: So, to confirm, you can install it stand alone but not buy it and use it that way. You have a stand alone install, but it requires an internet component to download. So... the product CANNOT be used on a stand-alone computer for sync.

Randy: Yes, you are right. May you have more questions for now?

Akiva: Well that's just frustrating. Kind of makes the USB sync feature useless. Oh well, thanks anyway.

So what happened here? This company developed a non-network based sync option, then later developed an internet based licensing requirement that invalidates the non-network based sync option – basically making the product useless. I wonder if they actually sell any of these, or sell them and have people demand refunds since it effectively is useless now. Bad planning, no overall architecture to understand the impact of one feature set on the rest of the system.

Bad Integration by Design or How to Make a Horrible Web Service

2014-12-25T17:22:00.001+02:00

To understand what makes easy integration or a “good web service”, it’s worth taking a glance at the historical methods of I.T. systems integration. After all, business systems have been passing data around and/or activating each other, aka integrating, for almost as long as there has been commercial I.T. business systems (approximately since 1960).

The first major “interface” method between systems was throwing sequential fixed-length record files at each other. This was pretty much the only method for 20 years and still remains in widespread use, though mostly around mainframe and legacy systems. The system providing the interface, either outputting the data or providing a format for which to send it data, defines a field by field interface record, along with header and footer records. Because these are fixed length records, the descriptive definition (the human readable documentation) must include the format and length of each field, along with any specialized logic interpretation or encoding. For example, if a record represents a person, which includes their gender, it might specify a 1 byte single digit field, with a 0 representing male and a 1 representing female. (Given that this appropriate started in the early days of computing, there is also a strong tendency to minimize data size – save the bytes! – leading to additional encoding logic within the definition.) Because the definition is fixed length records, no data typing can be enforced within the data format, only at time of programmatic interpretation.

So how did this approach work? It worked great. This is the base approach of generations of systems, especially financial and business systems.

If it worked great, why don’t we do this anymore?

Answer: Because of the data typing (no enforcement in the format), the encoding (no enforcement in the format and not understandable without documentation), and other dependent logic (such as cross field validation instructions, example “if field 2 is female, then you may fill out the field 9 for number of pregnancies”), getting an interface build and correct would take 2-6 weeks per connection. So while this method worked, it was time consuming to successfully implement.

API’s came along to allow direct activation, and defined a fixed set of data types required to activate. This solved the problem of the first model’s data typing without enforcement, and part of the documentation problem (the data types became self explanatory). Further, the API’s could define descriptive names for the data fields, thereby providing some self-documenting ability within the API. A major improvement.

API’s, however, added a new problem: they were technology, and often version, dependent. Meaning an API exposed on one system in one language in one release was compatible only with another system in a matching system / language / and version.

Regardless, integration via APIs was easier and faster. And it became the base technology that allowed Windows, Unix and other modern operating systems to move from being simply an execution starter and hardware interface to being a facilitator of interaction between applications. It further allowed a real-time interaction that was not possible previously. That said, figuring out and correctly using an API could still take days to weeks. Embedded cross field validation and logic would often slow down the process.

API’s evolved in the next generation with REMOTE APIs. Remote APIs moved the cross-application interaction to cross-system cross-environment interaction. The original remote API technology with commercial success included DCOM, CORBA, and RMI. All of these commercial implementations worked, but were very complicated and highly sensitive to perfect conditions. And, for the most part, they were TECHNOLOGY specific (as well as being version specific). So while they began to offer the new ability of remote invocation and/or coordinated system interaction, the environment had to be perfectly configured and matching technology and version.

Each one of these generations of integration technology worked within it’s context and solved problems not previously solvable – offering new abilities and new opportunities. Yet their limitations meant they remained niche solutions for specific narrow problems.

With the arrival of web services, a new integration level was reached. Web Services offered all the previous abilities while adding key points:

- The data format is XML, and therefore self descriptive.

- The service and data format is defined with an XSD, and therefore is self validating.

- The communication protocol is firewall and technology neutral and friendly.

- The data format is technology neutral and supported by all development tools.

With these abilities added to the historical ones, integration moved from a major project effort to…simple, trivial, fast. And with that change web services and integration became more than just commonplace, it became the way to do things. (This brings some new problems, such as integration spaghetti and interconnection dependencies, but that’s a different discussion.)

So how do you make a horrible web service? Simply strip away one more more of the primary advantages it offers. Examples:

-- Serialize language specific objects into your web service as one or more data items. For example, serialize a .NET object into your web service. The result, a web service that can only work with .NET (and of the appropriate version). Yes, I’ve seen this done.

-- Place “codes” in data fields in the web service. For example, make a field <Gender> where “3” = Male and “1” = Female. Then explain to the user of the web service that they must download your table of codes / values to insert the correct values or interpret the values. This, sadly, is a not-uncommon error.

-- Structure the XML as just a flat list of fields even though it could be placed in a hierarchy, or is in a hierarchy in the objects or database tables. The corollary of this error is to expose multiple services for each level of a hierarchy rather than one service with a hierarchy. This is the error of sharing data and not the business function / transaction. All too common.

In general, by stripping a web service approach to an earlier generation by stripping an ability, the result is a service of limited use, difficult re-use, and challenging to understand. Each of these problems turns into extra time and complexity, the exact opposite of what services come to solve.

I recommend avoiding these errors.

CIO Interview–Integration (SOA / SOAP / Web Services) Impact

2014-10-21T18:02:00.001+03:00

Some years ago I interviewed a CIO of a Fortune 500 IT vendor as part of an Integration Improvement project. His responses helped shape the goals and roadmap of the project, as business drivers and goals should always be taken into account in how one models the architecture and integration space. The interview gives a great view of IT management business drivers in the integration space. Company identifying information has been removed.

Question: How is integration?

-- Almost everyone at our company is an expert on Information Systems.

-- Our culture - everyone (department/division) believes their own numbers and requirements and doesn't believe anyone else's. Words like "mutual understanding" or "common terminology" didn't exist in the past.

The reason there is no real enterprise integration because of how the company is running. Each unit built their own systems.
The other units came to IT and said "buy this for us or build this for us, exactly per our requirements".
Result, we have a large number of systems, majority home grown because building to our own specs was (considered) easier (in the past.)

Today we want to do the opposite. We want to create best practices and show to our customers "this is the best way to manage your business". To implement the best practice by picking the best packages (vendor software) that does those processes. (Best of breed approach - but the business idea behind it is best-of-breed to implement specific target business processes, some of which may be offered as solutions to customers.)

Now we are not "developing" but coming with a solution. But there is no one company that can bring the best solutions across all the lines of business. Therefore we understand that we'll need multiple vendors to provide all the needs.

We have a variety of lines of business. And there are different requirements in the different lines. So we have to come up with the best practices for each business unit...and they don't want to bother with integration (they don't see integration are part of their business goals - interoperating with other business units is not a focus for them). But they want us (IT) to build them one integrated view (across business units) of what's going on.

The result - different packages (that we didn’t build), each from a different vendor. Some according to standards, some not [because sometimes the best solutions aren't standards compliant) and sometimes even different solutions from the same vendor are operating per different standards (a result of many vendor portfolios being built via acquisition).

The expectation is to build a common integration layer that can integrate the various tools from the different vendors….AND provide a common process view and data view that we can analyze.

...and the cost of maintaining this should go down (in comparison to today) and should continue to decrease.

So adding new systems shouldn't create a linear growth in the cost. The company wants to measure IT...what is the cost according to what it should have been by adding more and more systems? And how does it compare to the current cost of today. They want us to reduce the real maintenance cost of today even though we're adding systems and capabilities.

They want IT costs NOT TO GROW but provide additional capabilities (do more, spend the same).

How to do it? Some kind of miracle (we have to find).

Question: What's the time frame to meet these goals.

Answer: No one expects us to come up with all of this, this year. But what they do expect is new systems work should be meeting the new goals.

ROI - people have to commit to improved project impact by showing over 5 years how costs in particular areas (for example integration support) will be reduced. In order to make decisions we need to quantify everything as best as possible...even knowing some are assumptions that you can't really quantify at this stage. So we need to build a strict ROI but we can, for example, quantify values of agility (an example I provided).

Because we're in the process of replacing many older systems, we can easily quantify major ROI returns much faster. (Meaning switching old systems to a new method might have a very long ROI period, but incorporating the new method with the implementation of a new system eliminates a "change over" cost as that's already part of the old system replacement cost.)

After we make a decision on best internal practices, the cost of new projects will automatically include the overhead of doing the interface according to the right method/pattern/etc. And it's our job to convince the managers that it has the long term value in reducing the maintenance costs.

Question: Cultural drivers?

Answer: Our main driver, being able to respond to customers needs and changing business. As a company operating in technology fields, the technology is constantly changing as is the business environment.

We must be able to integrate new things very fast, and do so without destabilizing existing systems or significantly increasing support costs.

Today the capabilities we (IT) give to the company are very low. We're not meeting the business requirements of new functionality the business demands. There is a long back log of features the business has requested that we can't provide within our available budget and resources. We need to be able to delivery fast and agile.

How an Open Data Feed changed Israel’s Civil Defense

2014-08-20T19:21:00.001+03:00

Israel is under frequent, as frequent as every 10 minutes, rocket attack from Gaza. While the Iron Dome rocket interception system has become famous as a technological marvel in the defense of the country, there are other technological marvels of note as well.

The first step of any civil defense system is getting the civilians out of the way or under cover. Israel has a network of neighborhood bomb shelters, building bomb shelters, and (in new construction) a “hardened room” in every private residence and on every floor of every office building. When air raids were measured in hours or tens of minutes, all of these were adequate together with a nationwide network of air raid sirens.

But modern circumstances have brought two new problems:

- Rocket attack warnings are measured in SECONDS. Fifteen seconds in towns near border regions, and 90-120 seconds in the center of the country.

- As the country has suffered suburban sprawl, built malls and cinema mega-plexes, modern skyscrapers, joined the problems of traffic jams, and built it all with modern climate control (meaning sealed or closed windows and A/C), HEARING alarm sirens has become a problem.

Like any government agency, the Israeli Civil Defense department – a division of the Israeli army – has implemented big project approaches to these problems. A radio based pager like messaging device… too expensive except for large businesses or office buildings (which can then manually alert tenants). A cell based pager messaging device with digital output… with reception problems and a complicated interface requiring special software – again making it of limited use. The newest addition, SMS messages to all cell phones from cell towers in an alert area, a massive project that required integration with all the cell phone providers but only results in a regular SMS “ding” – making it useless.

A lot of effort and a lot of money with the problem continuing to grow and the current solutions offering only limited impact.

But then something amazing happened. The Civil Defense department public web site integrated a real time alert box onto the site. It was unnoticed by almost everybody except for a young man in southern Israel in a community frequently targeted. Since a web page is, by nature, open source, he looked into the page to determine where they were getting their data – their real time data of civil defense alerts for Israel.

He took the data feed, a nicely formed JSON data URL, set up a server polling it, and built an Android client. This became the first “Code Red Israel” alert app. Someone else contacted him and asked to use his server, and built an iPhone edition. This was 2 years ago.

With the current conflict and the terrorists expanding their targeting to civilian cities and towns across Israel, the apps gained notoriety. But so did the people interested in creating additional abilities, options, and clients. And an explosion in apps and abilities has been created over the past two months.

Examples include: real time alert monitoring web pages (in Hebrew and English), extensions for Chrome, iPhone and iPad apps that offer various sounds, filtering by city, maps of alert locations, commenting to share thoughts of being targeted, Android apps to do all of the same – in Hebrew, English, or Russian (major languages used by segments of the population in Israel). And like any app category, there’s become a competition between apps on offering the most useful features – even though most of the apps / pages / extensions do not charge or even offer ads (meaning they’re covering their development and server costs out of pocket).

Today while waiting in line at a grocery store or sitting in an office, almost everyone’s phone will go off if there’s an alert – some for only the local area, some for the whole country (as each person prefers).

There’s a key additional point. The data feed seems to be providing the alert data seconds before the actual sirens go off. So it is possible to get the alert up to ten seconds before the sirens actually go off! (Depending on the speed of the monitoring service.)

So while the Civil Defense department spent years and millions on building up a technological infrastructure, their biggest success was by accidentally offering an open JSON data feed.

My personal alert project is a web 2.0 site at http://IsraelSirens.com.

MDM & SOA - Layer, Repurpose or Replace?

2014-07-18T01:50:00.005+03:00

An Architect Friend sent me this extended architecture question...

I recently joined a company that provides business consulting (via many MBAs) services related to sales and marketing. Most clients are large pharma companies.

In addition to consultants, there are business process outsourcing teams (offshore) that do operations (like incentive plan management, report distribution, etc.) There is also BI/reporting group that creates BI/DW solutions (custom ones using template approach) for large clients. Plus there is an Software Development group (SD).

Over the years the Software Development group of the company created various (10+) browser-based (.NET/SQL) point-solutions/tools to help consultants (and eventually some head-quarters users) perform specific tasks. For example:

- Designing sales territories and managing the alignment of reps to territories
- Custom ETL-based tools to perform incentive calculation
- Some Salesforce-like platform for creating custom form-based apps

The applications are architected as single-tenant – with some deployment tricks to be able to deploy an “instance per client” on the web servers. The databases are isolated per client/instance. The tools are sold as if they are part of an integrated suite, but they aren’t natively integrated and require custom integration.

There is a custom grown ETL-like tool for interconnecting the tools to each other (but not standard connections since the data models are all “flexible” and not well defined) plus Informatica and Boomi to get data from clients. Some clients use one tool, some use 2, some use 3, etc. Some tools are used directly by the client, but most are used by the consulting teams on behalf of the client.

Lately, there is desire to make it all “integrated” across the company (SD + BI + all else). Two main themes are emerging (even prior to me joining): “common data model” and “SOA”. There is also the question of letting existing applications function as-is and developing new ones on a more proper architecture versus trying to evolve the existing apps.

However, the understanding of how this applies to an Enterprise looking inward on its own systems and trying to align them, versus Independent Software Vendor (ISV) looking to build software for other Enterprises did not yet sink in… and concepts are being confused…

The tension between a standardized productized software versus customized (consulting company) software solution is not yet resolved.

I wanted to ask if you had experience in environments were an ISV was trying to define the enterprise architecture of their solutions for customers versus their own internal architecture.

Are there any case-studies or resources you could point me to get some reference architecture examples?

I usually do not like “next gen” approaches, but I am not seeing much potential in evolution of existing assets into an integrated state (they have a lot of “baggage” and features that were there but don’t play nicely with “integrated” world-view).

=============

Here's my answer:

I wanted to ask if you had experience in environments were an ISV was trying to define the enterprise architecture of their solutions for customers versus their own internal architecture.

- No, though I have built integration competency centers and projects that were providing service environments across very large scale enterprises of disparate divisions.

Are there any case-studies or resources you could point me to get some reference architecture examples?

- Not that I know of. I'm not much of a fan of such studies, mostly because the requirements and details are always highly complex, and those details directly affect the approaches taken. Studies and reference architectures provide a nice high level structure – but the more you try to keep to them in the details the less effective they are (as they are mismatched to the exact situation). I use bits of Togaf-9 from opengroup.org, bits of CBDI from Everware http://www.everware-cbdi.com/, and various tidbits picked up from Zapthink (though every few years they discuss the benefits of yet another framework).

- It's a pretty standard problem: how to balance between what is, how it can be extended / expanded / reused, and what should be replaced / redeveloped / moved up to a new generation of technology, pattern and features.

- The problem you describe sounds like it crosses between SOA / integration and MDM (master data management). Sometimes a SOA façade can provide an MDM operational model, with composite services doing multi-system queries, combining or rationalizing the result, and presenting single meaningful "views". In other cases it's the SOA abilities enabling MDM to do it's job, which often involves signification bi-directional synchronization.

- The MDM tools tend to be heavy, and the business and systems analysis work (which system wins when data is in conflict, for example) is a major portion of the success or failure.

- That said, IF you are only trying to get views of the data, I am hearing reports of good success with some of the easier BigData tools (such as MongoDB). Success meaning they are able to develop and deploy meaningful business results in months, whereas MDM and big integration SOA projects almost always take over a year.

NSA Tapping and Cloud Computing

2013-08-26T20:53:00.001+03:00

Recent revelations from the U.S. have informed the public that conspiracy theorist fantasy's are all too real, the U.S. National Security Agency (NSA) has been installing taps at key Internet points to absorb vast quantities of email and Internet traffic.

As IT professionals, publically available information (no inside information or secret information is utilized in the preparation of this article) and an understanding of the Internet as a series of routers and servers, we understand that “taping key Internet points” means copying streams of Internet traffic via router configurations and having monitoring software installed on email servers and the like (directing copies to the government monitoring servers).

The NSA isn’t secretly tapping into some sort of vast Internet cable bundles. Rather they’re walking into AT&T, Google, Yahoo, Microsoft, Internet backbone and primary service providers, and installing software on their routers and servers as well as installing NSA receiving servers on their networks and premises.

A few months ago ZapThink published an article questioning why the move into Cloud Computing via Public Cloud Vendors has been relatively slow…

Cloud Computing: Rethinking Control of IT : Jason Bloomberg, April 24, 2013

In my role as a globetrotting Cloud consultant, I continue to be amazed at how many executives, both in IT and in the lines of business, still favor Private Clouds over Public. These managers are perfectly happy to pour money into newfangled data centers (sorry, “Private Clouds”), even though Amazon Web Services (AWS) and its brethren are reinventing the entire world of IT.

Their reason? Sometimes they believe Private Clouds will save them money over the Public Cloud option. No such luck: Private Clouds are dreadfully expensive to build, staff, and manage, while Public Cloud services continue to fall in price. Others point to security as the problem. No again. OK, maybe Private Clouds will give us sufficient elasticity? Probably not. Go through all the arguments, however, and they’re still dead set on building that Private Cloud. What gives?

The true reason for this stubbornness, of course, is the battle over control…

Why do (IT) executives crave control so badly? Two reasons: risk mitigation and differentiation. If that piece of technology is outside your control, then perhaps bad things will happen: security breaches, regulatory compliance violations, or performance issues, to name the scariest.

(The article continues why this isn’t really true and mitigated through SLA agreements and gives you the advantage of separating the responsibility and control.)

ZapThink misses the major point in my mind.

There’s been a variety of Cloud Computing articles superficially discussing potential legal complications of a corporation having some of their business data in a different legal or national jurisdictions from their business. And we’ve seen some practical challenges of major US service providers being hit with EU privacy standards violations (for example). What, for example, would happen if a customer sued the Cloud Vendor for “deletion rights” (an EU data right) for a EU customer who had signed up with a US Internet service provider that happened to use an EU based Cloud Resource Vendor for their storage? IT executives naturally shudder at the business / legal complexity of data crossing state / national / international borders.

With the NSA monitoring revelations, we see much worse concerns.

In the case above, I could end up in a lawsuit outside my jurisdiction. If I’m a small company, this could be catastrophic (if I’m a large corporation, only ridiculously expensive). But at least all I have is a legal risk. As long as the Cloud Resource or Platform Vendor is living up to their contractual responsibilities and technical features, my data and computing results remain private and controlled – though now I have the additional party of the Cloud Vendor in the mix.

With NSA monitoring, the Cloud Vendor may be forced (or coerced or tempted with financial payments) to provide monitoring access without being permitted to notify me (fully legal). My company would thereby have no legal recourse to attempt to protect our data because we wouldn’t even know such monitoring is occurring. (In instances of the monitoring revelations, it’s also become clear that the NSA is doing so with “secret court” orders which prevent the service providers / vendors from letting anyone know such monitoring is being requested or occurring.)

Suddenly the controlling or paranoid IT executive is looking smart.

We now have established the fact that if your data leaves your premises, it may be secretly tapped / copied / monitored, and you’ll likely never know (thereby offering no legal recourse to challenge it). And while we certainly want our national security resources to be able to do their jobs and provide national safety, we also know that such authority is subject to misuse – and misuse of key company data could cost millions or even billions, or put a company out of business.

When police or investigative authorities arrive with a court order, we may legally challenge it as well as trying to keep the access as narrow as possible. Even further, it may be our IT people providing the data (so we know exactly what’s leaving and what the potential business impact is). If the NSA is monitoring or accessing Cloud Vendors, our data is leaving without any control or even knowledge on our part. Our business risk is potentially unlimited.

The advantages of the Cloud now carry a real risk.

For personal use, cloud services now carry the same risk. If you’re tying your Android phone to Google account sync or your iPhone to the iCloud, your contacts are now (probably) being monitored. How about if you’re using a Cloud backup service or online file storage (Googe Drive, Microsoft SkyDrive, etc)? We don’t know, but with recent revelations, we’d be foolish to assume the data is being kept private from national security authorities – which included revelations that NSA employees used the service to spy on personal love interests.

If it leaves your premises and it’s not encrypted and kept encrypted at the destination, it’s only appropriate nowadays to assume it’s being monitored. And if it is encrypted, you may warrant special attention.

Welcome to the digital age. Your government is now online.

The Reality - Email Privacy or the Lack Therefore

2013-08-20T15:02:00.001+03:00

Today a top law blogger freaked out after realizing that their emails can be read and monitored...

"The owner of (a encrypted protected email service that just shut down) tells us that he's stopped using email and if we knew what he knew, we'd stop too. There is no way to (blog) without email. Therein lies the conundrum. What to do?

...the simple truth is, no matter how good the motives might be for collecting and screening everything we say to one another, and no matter how "clean" we all are ourselves from the standpoint of the screeners, I don't know how to function in such an atmosphere.

I feel (unclean), knowing that persons I don't know can paw through all my thoughts and hopes and plans in my emails... They tell us that if you send or receive an email from outside the US, it will be read. (And many emails inside the US are accidentally picked up by those capture engines.) If it's encrypted, they keep it for five years, presumably in the hopes of tech advancing to be able to decrypt it against your will and without your knowledge.

I hope that makes it clear why I can't continue. There is now no shield from forced exposure. …no one can feel protected enough from forced exposure any more to say anything the least bit (controversial or security related) to anyone in an email, particularly from the US out or to the US in, but really anywhere. You don't expect a stranger to read your private communications to a friend. And once you know they can, what is there to say?"

Much of the Internet and the abilities we take for granted today were never conceived to be the large world-wide network they have become today. Email, as used today, and it’s base protocol (SMTP) are not encrypted and follow the normal network routes to get where they are going. The term “email” is a mistake, because it’s NOT a sealed letter – it’s a open postcard (an open sheet of paper with an address on it).

This means:

- The sending post office and receiving post office can and do read the sending and received address AND the full content.

- Every network it passes through along the way (remember, the “Internet” is all cross connected networks”) can also capture and read the sending and receiving addresses AND the full content. (Today the average number of networks things pass through are 8-16.)

- All the little people along the way that operate all the stuff that makes this happen, such as system administrators, database administrators, network administrators, have all the tools in front of them every day as part of their jobs to read any of this stuff they want to.

That’s been the case from day 1 with Internet email. NO public free email service NOR any service offered by any Internet ISP encrypts email traffic or storage. It’s all traveling and sitting on open unprotected readable pieces of paper (so to speak). The ONLY protection has been “privacy policies” of the companies and them being shamed (and losing customers) if they didn’t provide a reasonable semblance of isolation of your emails.

Even so, YOU have voluntarily given them the right to “paw through your data” since day one, and much worse!

- Your emails are being automatically scanned by Google / Yahoo / Microsoft Hotmail-Outlook.com, "to serve you ads and understand their customers." Note the ads displayed on the email pages are context sensitive to the email being read, this doesn’t happen by magic but by your email being auto-scanned for keywords.

- Your phone location (meaning your body’s location) is being sent to Google and Apple, "to provide better mapping services and location based app responses (letting an app know where you are to tell you about something near by)." You can’t turn on GPS / Location Services without giving them permission to track you moment by moment.

- If that’s not enough, the cellphone service provider logs every call, who you called or who called you (number on both ends – which you see on your phone bill) AND the approximate location of your phone when the call began (tracked by which cell tower you connected to and the strength of your signal, meaning how far from the tower you are – combined with where the other towers are near that one this can be used to narrow your location to within 1/10 of a mile).

- Of course, every SMS is logged (source and destination) as well.

- Your office computer is tracking what web sites you browse, in some cases what programs you are running, in some cases even how fast and what you are typing, and reporting it to your IT department and/or your boss. If you are visiting whatever your office considers improper sites, the IT department and boss are being alerted.

- Your office emails are being scanned for improper words and phrases (sexual, harassing, racist, violent, threatening), alerting the IT department and your boss if there's a hit. They may also be scanned to see if you are sending out company proprietary information.

- Anytime you access any Google or Yahoo or Microsoft service, or for that matter any web site that chooses to collect the data, they know from where you did it (by IP address – which if correlated with the ISP will give a physical location down to where the connecting router is or actual cell location) and from what type of equipment you did it. Are you at home or in the office, on a desktop workstation, laptop, iPad or Galaxy phone, running Windows or a Mac or iOS or Android, which browser, and what size screen. This information can be obscured (with special fully legal utilities) except for the IP address, which can be via a VPN or anonymous service (also fully legal). But few people do so (and because few people do so, there’s a suspicion about people who do.)

Nothing is new with any of this. These are all a basic part of the operating model of all of these services from day one. If you wish to avoid it, you have to avoid using these services. (There are a few alternatives that theoretically offer protection, privacy or lack of logging – naturally such services are costly.)

However, there has been a FAÇADE of privacy and protection. And there has been a general thought that given the huge amount of such data, there may be anonymity and protection through obscurity. The façade has always been false – the data has always been easily accessible. And if it exists, police or courts or other authorities will go after it if it’s in their interest to do so. (There are companies who intentionally limit their logs of such things to 2 weeks or 3 months, to prevent them being called into any court case.)

The big deal now is that the U.S. government has joined the game and is, via various means, grabbing or monitoring some of the data above. And with new levels of computer processing power and “BigData” huge data set analysis techniques, Big Brother isn’t occasionally poking in to take a look – he’s constantly monitoring.

Given that the companies involved have basically being doing the same since day one, the primary change is that the façade has been blow away.

There are ways to reduce exposure without completely eliminating use of such services. But if you’re using Google or Microsoft or Facebook (or other such web sites or phone services), assume EVERYTHING you do is completely 100% public. Email, chat, SMS, sites you browse, pictures you share – it’s all being tracked … correlated, profiled.

Being we talk of BigData capabilities and BAM – Business Activity Monitoring, we in the IT industry shouldn’t be surprised.

How Granular? Or, What’s a Business Service?

2013-07-02T12:27:00.000+03:00

What is a Service or Component? It sounds like a simple question. But I was approached by an organization that had done no component or service oriented or even object oriented development. So the question was much more complicated…

What should a service / component / object be? What amount of business functionality should it include? Or, in technical terms, what granularity to your business services or components?

This question came up again for me, as I’m now dealing with the same question relative to Business Processes (BPM or BPA workflows). That will be a future post.

But for how granular should a service or component be? Here’s my presentation on the topic…

What is a SOA Service - from the Business Perspective? How detailed, what level of functionality does it expose? from Akiva Marks

Batch in the age of Java, SOA and Event Driven Business

2013-06-24T13:22:00.001+03:00

My customer presented me with a major enterprise architecture task. “We’re building new systems in a new environment, it’s time to modernize our batch processing. Please recommend a toolset and processing model to move our legacy batches to our new environment.” I didn’t realize it at the time, but this was a trick question.

I spent an extensive amount of time investigating modern batch tools as well as Java development and batch. What I found was a very limited set of such tools, but more important as I consulted various architects I know around the world, the question of “why are you trying to modernize batch?” kept arising.

The question is less straightforward than it sounds. SOA, a component orientation, real time business and event driven business make batch exceptionally challenging, but more importantly much less relevant.

Attached is the result of my research, a whitepaper on a modern approach to batch in light of Service Oriented Architecture, Real Time processing, and a more Event Driven business approach.

Modern Batch Directions - Moving from Legacy Batch to Service Oriented - Component Modeled - Event Driven Batch...

Is the Change the Project?

2013-02-05T08:45:00.000+02:00

(Large Government Agency) has begun one of the largest government IT projects in (Country).

(Large Government Agency) has a strong desire to move into the future, provide more efficient and user friendly systems with web and mobile interfaces, and move from 2^nd generation code based systems (requiring over 150 maintenance programmers) to a 5^th generation rules and workflow based system.

Unfortunately failure to upgrade their software environment and approach in the past left them with not only 2^nd generation code and user interface (NATURAL by Software AG and 3270 green screens running in an IBM Mainframe environment) but with a 2^nd generation architecture. The mixed screen & processing logic, pure mainframe batch processing model and narrowly specialized data model was judged unable to be extended for use as a transaction engine or even be service enabled. Certainly no part of a 3270 green screen 80x25 character based user interface can carry forward into a web model.

As a result (Large Government Agency) made the radical decision to THROW AWAY their complete existing application set and REBUILD THEIR FULL APPLICATION ENVIRONMENT. Project (Rebuild) at (Large Government Agency) is a 10 year project to replace each (Large Government Agency) business application set with a newly analyzed, architected, and developed application. The first application set is scheduled to go live in January, 2014, and will replace the Customer Data Management system, the (Main Business Process A) system, the Payment Processing System, and will create new systems for (previously non-automated high-personnel-overhead business process), Electronic Document Management (moving (Large Government Agency) from paper and file driven processes to fully electronic processes), and a new (customer submission process) Website.

The project involves over 120 consultants working in the (nice new) Technology Park in (major city) near the (Large Government Agency) headquarters but purposefully not in it. The project is not just creating new software but working on modernizing and streaming business processes, where possible given the complex laws, regulations, court decisions and labor union limitations.

Building a modern application from scratch, the project management has decided to go with a totally modern approach, meaning: True Service Oriented Architecture (not just web services, modular self-contained business step components), Rules Based (FICO Blaze rules engine), Event Driven, Workflow Process Controlled (Software AG Webmethods BPM engine), Java and IBM Websphere Application Server Java Enterprise Environment development environment, and DB/2 10 with a Hybrid-XML Data Model for complex yet easily adjustable data structures designed for fast reading of full data sets for rules processing.

~5% of the citizens of (country) interact with (Large Government Agency) on a regular basis. Today it takes an average of 142 days and 5 visits to (Large Government Agency) offices for those citizens to complete their interaction. This project will make the process both easier, faster and less expensive for (Large Government Agency) and for the citizens.

Of course, projects of this scope have great risks and a history of failure. A constant customer feedback process, agile development with independent modular deliverables attempts to minimize the risk. A rigorous testing, stress testing, and user testing schedule will further reduce the risk.

Yet, even with these project steps to reduce risk, an organization that has avoided any major systems or application changes in a human generation faces enormous cultural challenges in accepting such an upgrade. Cooperation between the business and IT has been focused around providing service (as a utility), not around explaining and automating processes. The IT infrastructure teams are used to managing equipment capacities several factors behind the current generation. Managing the cultural challenges within the organization and within the IT organization may be even higher risk than the large development scope of the project.

That said, (Large Government Agency) has hired teams of top tier consultants and given the project a realistic time frame and sufficient budget to make it happen. And perhaps most importantly, management has realistic expectations.

(Large Government Agency) has a reasonable likelihood of step by step success. If dealing with change is difficult in most organizations, dealing with change IS the real project in this organization.

My Internet’s Too Fast

2013-01-29T17:48:00.001+02:00

Whether we’re integrating systems or building new SOA architected applications (with heavy cross-component communication that may be flowing between physical machines or across virtual machine clusters – which again drop down to separate physical machines), the network backbone makes a significant difference.

A good enterprise data center is running at least a 1gbit backbone, with portions or major interconnects at 10gbit. Further, those good network engineers are properly segmenting the network, which makes sure the heavy traffic patterns have maximum capability (and aren’t going to be slowed down by users streaming internet radio).

Between applications in the core of the data center, we rarely run into network capacity or network speed as our major performance problem. (Credit those network engineers.) This doesn’t mean the issue should be ignored, if some applications are integrated in a high speed query or transactional pattern, there can definitely be a major performance benefit of making sure their connection moves up from 100mbit to 1gbit or from 1gbit to 10gbit. Poorly architected SOA services and application integrations that have many granular transactions and/or heavy crosstalk can see a surprising performance benefit from a network speed upgrade or even just creating separate or multiple network channels (spreading the communications between them across multiple physical network ports).

Somewhat surprisingly, those easier options and solutions are becoming LESS viable with the move to virtual machine environments (and for mainframe integration), where one or a small number of physical network ports is shared among a large number of virtual machines or LPARs. In these cases, increased network speed (with associated faster ports) is the only option.

This is a lower level physical problem that many an integration or SOA team will miss in trying to diagnose integration performance problems.

Interestingly, I was brought to this topic by a recent Internet upgrade at home. The cost of high speed has become reasonable and it’s availability spread across wide areas. I upgrade mine in the past week and was surprisingly disappointed.

Doing a number of checks with SpeedTest.net, I found I could get from 50% - 90% of my purchased speed to test points in my nearby area. Outside my area the numbers are all over the place…

New York – 70%
Paris – 10%
London – 40%

What’s going on? First is a basic problem… how’s my router performance? When we were working with 2mbit and 5mbit Internet, putting a slow cheap processor in the home routers was sufficient. Working with 10mbit, 30mbit or 50mbit Internet needs a LOT more router CPU power.

Second, just like in the data center example above, IF the routers or network paths I’m going through to get to the destination are over 50% of their capacity, my performance is going to suffer. And having some Internet hosting accounts for private and family use, I can certainly note that my hosting providers have NOT upgraded the capacity of those servers. (They may even have only 100mbit network ports.)

What does it mean when the default “low” speed for home users is becoming 10-20mbit, with low cost options of 20-100mbit INTERNET, yet the servers were built to serve 100 users at 1mbit each?

It means I’m not going to get 100% of my purchased speed because the sites I visit can’t deliver it.

My Internet is too fast.

Query Service or Synchronized Data?

2012-09-24T17:00:00.001+02:00

- When linking two systems, with one system providing data to another system, the providing system's physical environment must be sized for the capacity of the requesting system and the reliability of the requesting system. Another way of saying this is that the providing system must meet or exceed the quality of service or SLA of the requesting system.

- In the example I was reviewing, the HR system was the base for the desired information and was sized for the user capacity of the HR department and the reliability impact of an outage of the HR department. If it is to be used as a real-time providing system for Department B, it's capacity must be increased from the HR department (10 users) to the capacity of the Department B user base, the main business area (3,000 users). It must also have it's redundancy increased to provide no outages.

- Alternatively, when there is a mismatch between capacity and reliability of the providing system and requesting system, the data may be effectively de-normalized by building a copy or synchronization mechanism between the systems – the providing system sending a copy of the needed data set into the requesting system for local use. This may also be appropriate if the systems operate in different security domains, in different networks or network segments where bridging is a problem, or where the systems are separate by geographic areas where network performance [speed or capacity] are an issue. It's also frequently required where a packaged application is the requesting system, as most packaged applications will only query the data in their expected local format and cannot be redirected to a web service or other remote query.

- One other concern in such integrations is the lifecycle of the information. Is it static, reference information, infrequently updated, frequently updated or transactional or computed? Frequently updated information has dangers of synchronization problems or being out of date, and transactional or computed items can only be queried from their source system.

We generally hear that de-normalizing, whether in the database itself or across systems, is a crime (across systems would make it an integration crime?) Yet the circumstances above can make it preferred or even necessary. This is not a bad thing when done for the right reasons. and in a reasonably reliable way. (Noting that every synchronization operation should have a full re-sync option that should be run periodically [monthly or yearly or as appropriate].)

Categorize or Search?

2012-09-20T17:46:00.001+03:00

SOA design time governance products come with a variety of methods to categorize the services or assets. Interestingly I’m currently working on a project involving a document management system, and the select document management tools comes with almost an identical selection of categorization methods. These include trees, taxonomies, and domains among others.

In some of the earlier SOA design time governance implementations I performed, we spent significant time working on the categorizations – trying to make the catalog and information easy to traverse for the various user categories that would encounter in. (In the case of design time governance, this might be analysts, architects, developers, QA, and IT management.)

We invariably found designing the categories and approaches took a tremendous amount of time, with every constituency having different ideas and requesting various adjustments to the approach. To some extent it became the never ending quest for the perfect structure, never to be found.

We saw this same pattern emerge and fail among the World Wide Web. For those with a little Internet history behind them, they may remember the early Internet Indexing + Search Sites, such as Lycos, AltaVista, Netscape and Yahoo. Each presented various approaches to indexing, categorizing and presenting the Internet in various taxonomies.

One day a newcomer arrived named Google who presented one simple function…search. Within a short time all the indexing and taxonomy sites were dead.

The way we do things is limited by our technology. When we’re doing things manually, our technology may be bookshelves or filing cabinets, paper index cards and human retrieval methods – or even human memory capacity. As we automate processes with newer technology, it’s perfectly normal to take the previous process and “enhance” it with the new technology – but the previous process still exists.

Once we truly understand the possibilities and capabilities of the new technology, we often supersede the previous process. In the case of cataloging our information, our models came from books, libraries, index cards, filing cabinets, etc. Even if we were storing our information in high speed relational databases, our access models were based on our previous process – catalogs, sorted indexes, grouped information.

Yet Google has shown us that model is significantly less efficient and of less value that a capable search.

When we’re modeling today’s software abilities, user interfaces and data organization approaches, search should be the first and primary approach. Categorization is just pushing the old much less efficient approach forward.

Mainframe Integration–NATURAL Web Services

2012-09-20T16:49:00.001+03:00

Software AG’s Natural language and development environment for the mainframe (IBM z/OS and CICS) offered many nice improvements over COBOL for mainframe software development. Software AG developed a nice market niche in the mainframe development tools market, and millions of lines of code were developed and continue to run till today.

Like IBM’s version of COBOL, the vendor has struggled to extend Natural’s lifespan and the life of the code written in it. Initially Software AG created a communication bridging tool (called EntireX), allowing bi-direction communication between the Natural environment and Web Services or MQ or Java or .Net.

But like IBM came out with “native” web services as part of CICS 3 and Enterprise COBOL, Software AG has web service enabled Natural. And web service enablement includes the handling of (simplified) XML.

From the code sample I was able to find, it appears that Natural IS NOT handling the building of the HTTP headers, requiring them to be manually added to the top of the XML document – a rather odd lack, but use of direct web services significantly simplifies integration into the environment.

Here’s the code sample I found, which was used as a base to build some internal test services by a Natural programmer I know…

Description :

This example demonstrates how to call a SOAP service from Natural, using the REQUEST DOCUMENT statement, it then parses the output with the PARSE XML statement and presents a formatted view of the response.

The code is cross-platform, it works on both mainframe Natural as well as OpenSystems / LUW Natural alike.

Service used: http://www.webservicex.net/globalweather.asmx
Input asked for by the Natural program: City, Country
Output: weather details for the selected City + Country, or an error response

==============================================

*
* more info about the services to call: www.webservicex.net.
*
* better cross platform solution
*
DEFINE DATA LOCAL
1 #REQUEST              (A) DYNAMIC
1 #RESPONSE             (B) DYNAMIC
1 #RC                   (I4)
*
1 #PATH                 (A) DYNAMIC
1 #NAME                 (A) DYNAMIC
1 #VALUE                (A) DYNAMIC
*
1 HTTP-STATUS-TEXT      (A60)
*
1 #ACTIVE_NAME          (A16)
1 #REPLY
2 CITYNAME            (A50)
2 COUNTRYNAME         (A50)
2 LOCATION            (A50)
2 TIME                (A50)
2 WIND                (A50)
2 VISIBILITY          (A50)
2 SKYCONDITIONS       (A50)
2 TEMPERATURE         (A50)
2 DEWPOINT            (A50)
2 RELATIVEHUMIDITY    (A50)
2 PRESSURE            (A50)
2 STATUS              (A50)

* ---------------------------------------------------------- XML-COPY-OF
1 #XML-COPY-OF          (A) DYNAMIC
1 #FIND-TAG             (A) DYNAMIC
1 #TAG-FROM             (A) DYNAMIC
1 #TAG-TO               (A) DYNAMIC
1 #TAG-END              (A) DYNAMIC
1 #AT_SIGN              (A1)
1 #EXCLAMATION_MARK     (A1)
1 #APOSTROPHE                 (A1)
1 #QUOTATION_MARK                 (A1)
* ---------------------------------------------------------- XML-COPY-OF
END-DEFINE
*
INPUT // 'CityName...:' CITYNAME
       / 'CountryName:' COUNTRYNAME
      // 'Example: Tokyo, Japan; Heidelberg, Germany'
*
COMPRESS
'<?xml version="1.0" encoding="UTF-8" ?>' -
'<SOAP-ENV:Envelope ' -
'xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" ' -
'xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" ' -
'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ' -
'xmlns:xsd="http://www.w3.org/2001/XMLSchema">' -
'<SOAP-ENV:Body>' -
'<m:GetWeather xmlns:m="http://www.webserviceX.NET">' -
'<m:CityName>' CITYNAME '</m:CityName>' -
'<m:CountryName>' COUNTRYNAME '</m:CountryName>' -
'</m:GetWeather>' -
'</SOAP-ENV:Body>' -
'</SOAP-ENV:Envelope>' INTO
#REQUEST LEAVING NO
*
* For more information about the service have a look at:
* http://www.webservicex.net.
*
REQUEST DOCUMENT FROM 'http://www.webservicex.net/globalweather.asmx'
WITH
    HEADER
      NAME 'Request-Method' VALUE 'POST'
      NAME 'Content-Type' VALUE 'text/xml; encoding=utf-8'
      NAME 'SOAPAction' VALUE 'http://www.webserviceX.NET/GetWeather'
    DATA ALL #REQUEST ENCODED CODEPAGE 'utf-8'
RETURN
    HEADER NAME ' ' VALUE HTTP-STATUS-TEXT
    PAGE #RESPONSE
RESPONSE #RC
*
IF #RC = 200
*
* cut out the response
#FIND-TAG := 'soap:Envelope/soap:Body/GetWeatherResponse/GetWeatherResult'
PERFORM XML-COPY-OF
*
* parse the response
PARSE XML #XML-COPY-OF INTO PATH #PATH VALUE #VALUE
    DECIDE ON FIRST VALUE OF #PATH
      VALUE 'CurrentWeather/Location/$'
        #REPLY.LOCATION := #VALUE
      VALUE 'CurrentWeather/Time/$'
        #REPLY.TIME := #VALUE
      VALUE 'CurrentWeather/Wind/$'
        #REPLY.WIND := #VALUE
      VALUE 'CurrentWeather/Visibility/$'
        #REPLY.VISIBILITY := #VALUE
      VALUE 'CurrentWeather/SkyConditions/$'
        #REPLY.SKYCONDITIONS := #VALUE
      VALUE 'CurrentWeather/Temperature/$'
        #REPLY.TEMPERATURE := #VALUE
      VALUE 'CurrentWeather/DewPointh/$'
        #REPLY.DEWPOINT := #VALUE
      VALUE 'CurrentWeather/RelativeHumidity/$'
        #REPLY.RELATIVEHUMIDITY := #VALUE
      VALUE 'CurrentWeather/Pressure/$'
        #REPLY.PRESSURE := #VALUE
      VALUE 'CurrentWeather/Status/$'
        #REPLY.STATUS := #VALUE
      NONE VALUE IGNORE
    END-DECIDE

END-PARSE
*
WRITE #REPLY.CITYNAME '-' #REPLY.COUNTRYNAME //
    'Location ............' #REPLY.LOCATION /
    'Time ................' #REPLY.TIME /
    'Wind ................' #REPLY.WIND /
    'Visibility ..........' #REPLY.VISIBILITY /
    'Sky Conditions ......' #REPLY.SKYCONDITIONS /
    'Temperature .........' #REPLY.TEMPERATURE /
    'Dew Point ...........' #REPLY.DEWPOINT /
    'Relative Humidity ...' #REPLY.RELATIVEHUMIDITY /
    'Pressure ............' #REPLY.PRESSURE /
    'Status ..............' #REPLY.STATUS /
*
ELSE
WRITE // 'HTTP-Status:' HTTP-STATUS-TEXT //
END-IF
*
* copy the content of a xml element
* works equal to a XSL copy-of
*
DEFINE SUBROUTINE XML-COPY-OF
* for the mainframe
IF H'41' EQ "A" THEN
#AT_SIGN := '@'
#EXCLAMATION_MARK := '!'
#APOSTROPHE := H'27'
#QUOTATION_MARK := H'22'
ELSE
#AT_SIGN := '?‚?§'
#EXCLAMATION_MARK := UH'0021'
#APOSTROPHE := H'7D'
#QUOTATION_MARK := H'7F'
END-IF
*
COMPRESS #FIND-TAG                    INTO #FIND-TAG LEAVING NO
COMPRESS #FIND-TAG '/'                INTO #TAG-FROM LEAVING NO
COMPRESS #FIND-TAG '/'-H'FF'          INTO #TAG-TO   LEAVING NO
COMPRESS #FIND-TAG '//'               INTO #TAG-END LEAVING NO
*
RESET INITIAL #XML-COPY-OF
*
PARSE XML #RESPONSE INTO PATH #PATH NAME #NAME VALUE #VALUE
DECIDE ON FIRST #PATH
/*( end tag
    VALUE #TAG-END
*
*     remove false xml header
      EXAMINE #XML-COPY-OF FOR PATTERN '<?xml*?>' DELETE
*
      COMPRESS '<?xml version="1.0" ?>'
       #XML-COPY-OF INTO #XML-COPY-OF LEAVING NO
*
      ESCAPE BOTTOM
/*)
/*( start tag
    VALUE #FIND-TAG
      IGNORE
*
/*( content
    VALUE #TAG-FROM : #TAG-TO
*
      DECIDE ON FIRST *PARSE-TYPE
        VALUE "?"
         COMPRESS FULL #XML-COPY-OF '<?' #NAME ' ' #VALUE '?>'
           INTO #XML-COPY-OF LEAVING NO
        VALUE #EXCLAMATION_MARK /* Comment.
          IGNORE
        VALUE "C" /* CDATA section.
          IGNORE
        VALUE "T" /* Starting tag.
          COMPRESS FULL #XML-COPY-OF '<' #NAME
            INTO #XML-COPY-OF LEAVING NO
        VALUE #AT_SIGN /* Attribute (or ?‚?§ on mainframes).
          IF #VALUE = SCAN(#APOSTROPHE) /* APOSTROPHE or QUOTATION_MARK
            COMPRESS FULL #XML-COPY-OF ' ' #NAME '='
              #QUOTATION_MARK #VALUE #QUOTATION_MARK
              INTO #XML-COPY-OF LEAVING NO
          ELSE
            COMPRESS FULL #XML-COPY-OF ' ' #NAME '='
              #APOSTROPHE #VALUE #APOSTROPHE
              INTO #XML-COPY-OF LEAVING NO
          END-IF
        VALUE "/" /* Closing tag.
          COMPRESS FULL #XML-COPY-OF '</' #NAME '>'
            INTO #XML-COPY-OF LEAVING NO
        NONE VALUE /* $ Parsed data.
          COMPRESS FULL #XML-COPY-OF #VALUE
            INTO #XML-COPY-OF LEAVING NO
      END-DECIDE
/*)
/*( none
    NONE VALUE
      IGNORE
/*)
END-DECIDE
END-PARSE
END-SUBROUTINE
END

Hints from the Grey Haired Programmers

2012-06-27T21:56:00.001+03:00

I attended a major vendor conference this week. This particular vendor has a a modern line of current modern products, and also a set of 2nd generation mainframe products. They’ve wisely bought a series of smaller product companies over the past decade to ensure their future.

But the old product set continues to live on as well. Yes they’re offering modern interfaces, web service enabling abilities and so forth. And naturally they’re still investing in the original product set, as a cash cow should be milked as long as possible.

One of the speakers I wanted to hear was speaking in the older product set track, so I made my way to that conference area. As I entered it a generational change occurred. EVERYONE, and I mean everyone, was 55 years old or older – with the majority seemingly very very close to retirement.

There’s a clear hint from this sight for those using this product set. We normally think of end of life for a technology as when either support ends or the supporting platform is discontinued. But clearly there are a variety of applications and tools that are tremendously outdated but still in operation.

These products continue as companies have millions invested in their use, customizations, development of code, and all the factors that go into making a system of value to a business.

But while these products may continue to live, there does come a true end of life stage. In the case I’m describing, it clearly seems the knowledge set of this product is literally leaving the industry as the people retire.

While many vendors are quite happy to extend the life of their older and even oldest products forever (for a high support fee), customers would be wise to look that the availability of skilled knowledgeable people to work with them. When the skilled people are reaching the end of their working life, clearly the products have to be replaced. Hopefully before skills availability diminishes to a severe level.