Christina の J老闆

Observability in Hybrid Multi-cloud environment

2021-08-12T10:48:00.006+08:00

Quick recap,

Last article I talked about the study I did among Red Hat customers that makes the jump towards deploying their workloads on hybrid and multi-cloud environments. These articles are abstractions of the common generic components summarized according to the actual implementations.

To overcome the common obstacles of going hybrid and multi-cloud, such as finding talents with multi-cloud knowledge. Secure and protect across low trust networks or just day to day operation across the board. I have identify some solutions from the study, where I will be covering in the serie of articles:

Customize monitoring and logging strategy,

One of the keys to manage multiple clusters is observability. In a single cluster environment, metrics and logs are segregated in different layers, application, cluster components and nodes, now adding in the complexity of multiple clusters on different clouds will definitely make it more chaotic than ever. By customizing how to gather and view the metrics allows you to be more effective, easier to operate, pinpoint and locate problems quickly. To do that, we will need to decide how we want to aggregate the collected data, by regions, operations or a centralized point. (Depends on your HA strategy and how much data you are collecting). And then decide how to scrape or collect the metrics and logs. Historical data are valuable not only for us to view and identify problems. Since many vendors now support AI based remediation, by validating and extracting the data for operational models. Therefore we will also need to persist data.

How it works,

Checkout my previous article on the setup of the hybrid and multi cloud environment, and if you are interested, other articles on GitOps and secure dynamic infrastructure. But this time, let’s look at how observability works. First to view things in a single pane of glass, we host Grafana dashboard to stream and query the Observatorium service in all managed clusters.

When bootstrapping the managed clusters on each cloud, we will need to install the following,

For Monitoring,

Prometheus is installed to scrape metrics for cluster components, as well as the applications. A Thanos sidecar gets deployed along with a Prometheus instance to persist metrics to storage and allow instant query to Prometheus data. You may have multiple Prometheus instance per cluster depends on how you want to distribute the workload.

Observatorium is installed based on your defined strategy for observability(in this case, per region), this will deploy a set of service instances on the cluster. Where they will be responsible for aggregating, storing input metrics, as well as efficiently storing the data. Also providing endpoints(API) for observation tools like Grafana to query the persisted data.

Queries from the Grafana dashboard in Hub cluster, the central Querier component in Observatorium process the PromQL queries and aggregate the results.
Prometheus scraps metrics in the local cluster, Thano sidecar pushes metrics to Observatorium to persist in storage.
Thanos sidecar acts as a proxy that serves Prometheus’s local data over Thanos’s gRPC API from the Querier.

For Logging,

Promtail collects log files with fine-grained control of what to ingest, what to drop, and the final metadata to attach to the log line. Similar to Prometheus, multiple Promtail instances can be installed per cluster depending on how you want to distribute the logging workload.

Observatorium in each defined region are also configured with Grafana Loki, where it aggregates logs by labeling each log stream pushed from Promtail. Not only it persist logs in stooges but also allow you to query the high cardinality data for better analysis and visualization.

Promtail is used to collect log and push to Loki API (Observatorium)
In Observatorium, Loki distributor sends logs in batches to ingester, where they will be persisted. Couple of things to beware both ingester and querier requires large memory consumption, will need more replicas
Grafana dashboard in Hub cluster display logs via requesting

Real time display(tail) with websocket
Time-series base query with HTTP

This summarizes how observability was implemented in our customer’s hybrid multi-cloud environment. If you want to dive deeper into the technology, check out a video done by Ales Nosek.

Overview of making Hybrid Multi-cloud GitOps works

2021-08-11T03:10:00.002+08:00

Hybrid multi cloud can be a difficult, this is my study of a real customer use case on their journey using GitOps, multi cluster management system and securing dynamic infrastructure secrets.

Quick recap,

IaC and GitOps,

Infrastructure as code (IaC) allows the infrastructure to be more visible among teams through sharable files and scripts that contain how environments are set up and configured. We can also automate to speed up the provisioning and deployment process and avoid manual misconfiguration. GitOps takes it a step further, better managing the setting and configuration files into a versioned, centralized repository as the single source of truth. This allows better collaboration. At the same time, applying pipeline along with status check and testing, is perfect for introducing the continuous integration and continuous delivery (CI/CD) practice. This eliminates configuration drift, by constantly checking if the end environment is in-sync with the desired state defined in the repository.

How it works,

First, we start with getting the management hub ready. (for more information, see my previous post.) In short, in the management hub, we will have Red Hat Advanced Cluster Management for Kubernetes (RHACM) installed. Where it is used for provisioning/patching/updating the OpenShift/Kubernetes clusters (This is the key to hybrid and multi cloud environments). On top of RHACM, OpenShift GitOps is also installed for declarative continuous delivery, where it watches the manifests in repositories and automatically(or manually after configuration) updates deployments to the desired state.

Here is the flow of infrastructure continuous delivery on hybrid and multi cloud:

Manifest and configuration are set as code template in the form of “Kustomization” yaml. It describes the end desire state of how the managed cluster is going to be like. When done, it is pushed into the source control management repository with version assigned to each update.
OpenShift GitOps watches the repository and detects changes in the repository.
OpenShift GitOps creates/updates the manifest by creating Kuberenet objects on top of RHACM.

ACM provision/update/delete managed clusters and configuration according to the manifest. In the manifest, you can configure what cloud provider the cluster will be on, the name of the cluster, infra node details and worker node. Governance policy can also be applied as well as provision an agent in the cluster as the bridge between the control center and the managed cluster. Here are some snippets of how we can describe the end state in the managed cluster.  

 
apiVersion: hive.openshift.io/v1
kind: MachinePool
metadata:
  name: ....
spec:
  .....
  platform:
    aws:
      rootVolume:
        iops: 100
        size: 100
        type: gp2
      type: m5.xlarge
  ...
  replicas: 3
Above describe the infrastructural node needed on AWS . 
 
apiVersion: hive.openshift.io/v1
kind: MachinePool
metadata:
  name: ....
spec:
  .....
  platform:
    azure:
      osDisk:
    	diskSizeGB: 128
      type: Standard_D2s_v3
  ...
  labels:
    node-role.kubernetes.io/infra: ""
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/infra
  replicas: 3
Above for Azure 
 
apiVersion: hive.openshift.io/v1
kind: MachinePool
metadata:
  name: ....
spec:
  .....
  platform:
    gcp:
  	type: n1-standard-4
  ...
  labels:
    node-role.kubernetes.io/infra: ""
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/infra
  replicas: 3
Last for Google Cloud Platform

OpenShift GitOps will continuously watch between the code repository and status of the clusters reported back to RHACM. Any configuration drift or in case of any failure, it will automatically try to remediate by applying the manifest (Or showing alerts for manual intervention).

For application or updating during the continuous delivery, the process is very similar:

Developers have continuous integration pipelines to build application binary and images. Any changes to the infrastructure will also be pushed to the source control management repository.
OpenShift GitOps watches the repository and detects changes in the repository.
OpenShift GitOps create/update kustomize resources directly on to the managed clusters via the control plane
Change locally in the managed cluster.

This concludes my study on the overall architecture of how GitOps works in Hybrid Multi-cloud environments. If you want to dive deeper into the technology, check out Ales Nosek’s youtube video, where in his video he will take you through step by step on how to do GitOps to manage your clusters.

Hybrid Multi-cloud dynamic security management

2021-08-10T22:29:00.004+08:00

Hybrid multi cloud can be a difficult, this is my study of a real customer use case on their journey using GitOps, multi cluster management system and securing dynamic infrastructure secrets.

Quick recap,

In my series of articles I went over the study I did among Red Hat customers that makes the jump towards deploying their workloads on hybrid and multi-cloud environments. These articles are abstractions of the common generic components summarized according to the actual implementations.

Dynamic Security Management,

Kubernetes offers it’s own secret management control, although it’s sufficient for running a single cluster, but when you are trying to manage multiple sets of credentials and secure configurations, especially with the introduction of automated process and continuous delivery practice. We need a better way to securely and centrally manage these data.

How it works,

Checkout my previous article on the setup of the hybrid and multi cloud environment, and if you are interested, another article on getting the GitOps works. But for now, we are going to assume we have a fleet of clusters deployed on top of multiple cloud vendors, and one in the local data center. All the infrastructure is set as code and stored in a source management system. Where our GitOps system constantly coverage the managed clusters with its desired state. In order to setup a secure way to manage credentials and configuration cross clusters, we need two components,

External Secret management in OpenShift/Kubernetes

Enable use of external secret management systems (like HashiCorp Vault in this case) to securely add secrets into the OpenShift platform.

Hashicorp Vault

Secure centralized store for dynamic infrastructure and application across clusters. For low trust networks between clouds and data centers.

This is how the two components work together to manage secret in dynamic infrastructure:

During setup, the token to securely access HashiCorp Vault is stored in Ansible Vault. It is encrypted to protect sensitive content.
Red Hat Advanced Cluster Management for Kubernetes (RHACM) allows us to have centralized control over the managing clusters. It acquires the token from Ansible Vault during install and distributes among the clusters.
To allow the cluster access to the external vault, we need to set up the external secret management. OpenShift Gitops is used to deploy the external secret object to a managed cluster.
External secret management fetches secrets from HashiCorp Vault using the token we created in step 2. And constantly watch for updates.
Secrets are created in each namespace, where applications can use.

This is how to manage dynamic infrastructure secret in a multi cluster and cloud environment.

A study of hosting and managing on Hybrid Multi-cloud

2021-08-10T02:34:00.008+08:00

Hybrid multi cloud can be a difficult, this is my study of a real customer use case on their journey using GitOps, multi cluster management system and securing dynamic infrastructure secrets.

The Intro,

More companies are strategizing to be on Hybrid cloud or even Multi-cloud, for higher flexibility, resiliency and sometimes, simply it’s too risky to put the egg in the same basket. This is a study based on real solutions using Red Hat’s open source technology. This article is an abstraction of the common generic components summarized according to the implementation. And give you an overall idea of the flow and why we choose certain technologies to set you off the right place to begin your own journey on Hybrid Multi-cloud environments.
The idea of distributed computing is not new, by leveraging the combined processing power, memory and storage of multiple software components on multiple machine instances to achieve better performance. The problem now is how can we scale out the deployment for these software components quickly among the clouds with stability of actual machines. Having the freedom to bring up clusters close to the clients issuing the request and close to the data stores due to data gravity. Or sometimes they want to deploy a section of the application supporting the cognitive services they were running on the specific cloud providers.

To host platforms on multiple clouds can be difficult, as it introduces extra complexity. No matter if it’s finding the people with knowledge on all cloud vendors, securing across clouds and governance across the board. We found the most common questions from customers are, automation, security and uniformity. I have broken down how this study tackles above concerns using Red Hat and it’s partner’s technologies into four sections:

The Overview,

We have logically separated it into three main areas,

Unified management Hub, which hosts the management platform to manage all custers, a vault securing and issuing infrastructure credentials, a repository that stores the infrastructure code. And a CI/CD controller which continuously monitors and applies updates. I found many customers decided to host the hub in their own data center on top of their existing virtualization infrastructure.

Managed clusters, these are the clusters that run the customer’s application, scaling up/down for distributed computing needs. Metrics and status are constantly synchronized back to the unified management hub. These clusters are deployed across major cloud vendors such as Azure, AWS and Google cloud.

Bootstrap automation, this is a temporary instance that is used for bootstrapping the unified management hub. It consists of multiple Ansible playbooks to install all the components on the hub, and also setup assigned the administrative roles.

The Technology Stack,

In the case study, customers have chosen several the following technologies and the reason why:

Red Hat OpenShift Platform
- Instead of directly using and learning the offering from all vendors, or even learning the subtle differences between the Kubernetes offering, using a platform offering sits on top across data centers, private and public cloud will provide an unified way to deploy, monitor and automate all the clusters.
- OpenShift GitOps
  - Automate delivery through DevOps practices across multicluster OpenShift and Kubernetes infrastructure, with the choice of either automatically or manually synchronizing the deployment of clusters according to what’s in the repository.
- Core Monitoring
  - OpenShift has a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. On top of that, we can also define monitoring for user-defined projects as well.
- Grafana Loki
  - Horizontally-scalable and better log aggregation system, and more cost effective and easy to operate especially in a multi-cluster environment.
- External Secret
  - Enable use of external secret management systems (like HashiCorp Vault in this case) to securely add secrets into the OpenShift platform.
Red Hat Advanced Cluster Management for Kubernetes
- Controls clusters and applications from a single unified management hub console , with built-in security policies, provisioning cluster and application lifecycles. Especially important when it comes to managing on top of multi-clouds.
Red Hat Ansible Automation
- Used for automate configuration and installation of the management hub.
Hashicorp Vault
- Secure centralized store for dynamic infrastructure and application across clusters. For low trust networks between clouds and data centers.

The Setup,

The key to automate is “Infrastructure as code”, by versioning and storing clusters, networks, servers, data stores or even applications as software into a centralized and controlled repository, it allows the environment to be agile, consistent and less error prone. As the creation, updates are all pre-configured, and can be applied by simply executing the code, with less human errors and can be replicated across different environments.

We will start by bootstrapping the management hub. Here are the steps

First, we need to setup the Red Hat OpenShift Platform (OpenShift) that hosts the Management Hub. By using the OpenShift installation program, it provides flexible ways to get OpenShift installed. Ansible playbook used to kick off the installation with configurations.
Ansible playbooks are again used to deploy and configure Red Hat Advanced Cluster Management for Kubernetes (RHACM) and later other supporting components (External secret management) on top of the provisioned OpenShift cluster.
Install Vault with Ansible playbook. The vault we choose is from our partner Hashicorp, the vault is to manage secrets for all the Openshift clusters.
Ansible playbook is used again to configure and trigger the Openshift Gitops operator on the hub cluster. And deploy the Openshift Gitops instance for continuous delivery.

For identity management, we use the existing one and use it as a source for openshift groups. And later use it to authenticate users when logging into the Hub and managed clusters.

Now we have the centralized unified management hub ready to go, we are now ready to deploy the cluster on multi cloud to serve the developers and end users. In my next article, I will go over my study on GitOps. And simplify provisioning or updating in the complex setting.

Make SAP Cloud Native and Event Driven in 4 days

2021-06-28T19:00:00.001+08:00

Recently I had an opportunity to work with Sanket Taur (IBM UK) and his team on a demo, showcasing how Red Hat products can help speed up innovation with SAP Landscapes. To be honest I was shocked at how little time we were given to create the entire demo from scratch. It’s less than a week. While still doing our day job, having a couple of hours per day to work on it. If this doesn’t convince you.. I don’t know any other stronger proof than this, to show how agile and fast a cloud solution can be from development to production.

I encourage you to attend Sanket’s session for more details, this blog is JUST my view on the demo, and things I did to make it running. The demo was a simple approval process of Sales Orders. The SOs are created in the Core SAP platform (In this case ES5), therefore we need to create an application that speaks to the Core SAP platform and retrieve all the data needed.

First thing first, we need a Kubernetes(k8s) platform. And then I used Camel K — an enhanced framework based on Camel (part of Red Hat Integration product) to create the application. There was some mixup during the setup, instead of the OData v4 endpoint from ES5 for SO, line items and customer details. I was given an OData v2 endpoint. (Needless to say, how more efficient the OData v4 is, compared to v2. Please do update it when you have a chance). Note that Camel K only supports OData v4. HOWEVER, we can still get the results using normal REST API calls (So you are still covered).

This is how Camel helps you retrieve all the information needed. As you can see I have made several requests to get all the data needed as well as doing some transformation to extract results to return.

from("direct:getSO")

   .setHeader("Authorization").constant("Basic XXXX")
   .setHeader("Accept").constant("application/json")
   .toD("https://sapes5.sapdevcenter.com/sap/opu/odata/iwbep/GWSAMPLE_BASIC/SalesOrderSet('${header.SalesOrderID}')?bridgeEndpoint=true")
   .unmarshal().json()
   .setHeader("CustomerID").simple("${body[d][CustomerID]}")
   .marshal().json()
   .bean(this, "setSO(\"${body}\",\"${headers.CustomerID}\")")

;

 from("direct:getItems")
     .setHeader("Authorization").constant("Basic XXXX")
     .setHeader("Accept").constant("application/json")
     .toD("https://sapes5.sapdevcenter.com/sap/opu/odata/iwbep/GWSAMPLE_BASIC/SalesOrderSet('${header.SalesOrderID}')/ToLineItems?bridgeEndpoint=true")
     .unmarshal().json()
     .marshal().json()
     .bean(this, "setPO(\"${body}\")")
 ;

 from("direct:getCustomer")
     .setHeader("Authorization").constant("Basic XXXX")
     .setHeader("Accept").constant("application/json")
     .toD("https://sapes5.sapdevcenter.com/sap/opu/odata/iwbep/GWSAMPLE_BASIC/BusinessPartnerSet('${header.CustomerID}')?bridgeEndpoint=true")
     .unmarshal().json()
     .marshal().json()     
     .bean(this, "setCust(\"${body}\")")

 ;

The endpoints to trigger the call to SAP, is exposed as an API. Here I use Apicurio Studio to define the API contract. With two endpoints, fetch and fetchall. One returns SO, PO and Customer data, where the other one returns a collection of them.

We can now export the definition as a OpenAPI Specification contract in the form of YAML (Link to see the yaml). Save the file into the folder of where your Camel application is. Add the API yaml file name to your Camel K application mode line, and Camel K will automatically map your code to this contract.

   // camel-k: language=java dependency=camel-openapi-java open-api=ibm-sap.yaml dependency=camel-jackson

By using the Camel K CLI tool. Run the command to deploy the code to the OpenShift platform.

   kamel run SapOdata.java

And you should now see a microservice running. Did you notice how Camel K helps you, not only it detects and loads the libraries needed for you, but also containerised it as a running instance.

Go to my git repo to see the full code and running instructions.

Kafka was used in the middle to set the event driven architecture. So the SO approval application can notify the shopping cart client when it’s been approved.

Since everything was put together in a week, with everyone in different timezones, miss communication will happen. What I did not realize was that all the client applications, SO approval and shopping carts were all written in JavaScript, and must communicate via HTTP. But Kafka only does Kafka protocols!!! Therefore, I set up an Http Bridge in front of the Kafka clusters, so it will now translate the Kafka protocols.

And now clients can access the topic via HTTP endpoints. For more information on how to set, go to my Github repo for more detailed instructions.

Last but not least, we need to migrate all UI5 SAP web applications to OpenShift. The UI5 is basically an NODEJS app. We first create the docker file to containerize it. And push it to a container registry.

  docker push quay.io/<YOUR_REPO>/socreate

And deploy the application to OpenShift.

  oc new-app quay.io/<YOUR_REPO>/socreate:latest --as-deployment-config

BUT WAIT!! Since UI5 only does binds to *localhost* (weird..), we need to add a proxy that can tunnel traffic to it. Therefore, I added a sidecar proxy running right next to the NodeJS application. By adding the following configuration.

spec:
      containers:
        - name: nginx
          image: quay.io/weimei79/nginx-sidecar
          ports:
            - containerPort: 8081
              protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 1Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always

This will start the proxy, and since this NGINX proxy starts on port 8081, make sure you update all related settings on OpenShift.

  oc expose dc socreate --port=8181
  oc expose svc socreate

And this is how you would migrate the UI5 application from a local SAP instance onto OpenShift. More detailed migration instructions, check out my Github repo.

Once it’s done, you can see all the applications are running as a container on the cloud. And ready to approve the SOs.

This is actual developer view on top of our demo OpenShift platform

Thank you Sanket for this fun ride, all the nail biting moments, but this is all the fun in IT right? We work through problems, tackle issues and ultimately get everything done! 🙂 If you are a SAPer, and want to join the container world of cloud, what are you still waiting for? Join the ride! This is the story on how we made SAP Cloud Native and Event Driven in 4 days.

To see the full version, be sure to attend Sanket’s session:

SAP & OpenShift: From classic ABAP development to cloud native applications: Use cases and reference architecture to implement with SAP Landscapes to unlock innovation enabled by Hybrid Cloud and Red Hat OpenShift.

https://events.redhat.com/profile/form/index.cfm?PKformID=0x373237abcd&sc_cid=7013a000002w6W7AAI

Kamelet for streaming to Kafka!

2021-06-21T22:10:00.002+08:00

You want Kafka to stream and process the data. But what comes after you set up the platform, planned the partitioning strategy, storage options, and configured the data durability? Yes! How to stream data in and out of the platform. And this is exactly what I want to discuss today.

The Background

Before we go any further, let’s see what Kafka did to make itself blazing fast? Kafka is optimized for writing the stream data in binary format, that basically logs everything directly to the file system (Sequential I/O) and makes minimum effort to process what's in the data (Optimize for Zero Copy). Kafka is super-charged at making sure data is stored as quickly as possible, and quickly replicating for a large number of consumers. But terrible at communication, the client that pushes content needs to SPEAK Kafka.

Here we are, having a super fast logging and distributing platform, but dumb at connecting to other data sources. So who is going to validate the data sent in/out of the kafka topic? What if I need to transform the data content? Can I filter the content partially? You guessed it. The clients. We now need smart clients that do most of the content processing and speak Kafka at the same time.

What are the most used connect tools today for Kafka users?

Kafka Connect is what the majority of the Kafka users are using today. It has been broken down into many parts such as connector, tasks, worker, converter, transformer and error handler. You can view the task and worker as how the data flow is executed. For a developer they will be mostly configuring the rest 4 pieces.

Connector - Describes the kind of source or the sink of the data flow, translating between the client/Kafka protocol, and knowing the libraries needed.
Converter - Converts the binary to the data format accepted by the client or vice versa (Currently there is limited support from Confluent, they only do data format) And does data format validation.
Transformer - Reads into the data format, can help make simple changes to individual data chunks. Normally you would do filtering, masking or any minor changes. ( This does not support simple calculations)
Error Handler - Define a place to store problematic data (Confluent : Dead letter queues are only applicable for sink connectors.)

After configuring, it then uses Task and Worker to determine how to scale and execute that pipe data in/out of Kaka. For instance, running a cluster of works to scale and allow tasks to perform parallel processing of streams.

Camel is another great option!

Apache Camel is a GREAT alternative for connecting Kafka too. Here’s what Camel has to offer.

Connector - Camel has more than 300+ connectors, you can use it to configure as source or the sink of the data flow, translating between the 100+client/Kafka protocol.
Converter - Validate and transform data formats with simple configuration.
Transformer - Not only does simple message modification, it can apply integration patterns that are good for streaming processing, such as split, filter, even customization of processes.
Error Handler - Dead letter queue, catching exceptions.

There are also many ways to run Camel. You can have it running as a standalone single process that directly streams data in/out of Kafka . But Kamel works EXCEPTIONALLY well on Kubernetes. It run as a cluster of instances, that execute in parallel to maximize the performance. It can be deployed as native image through Quarkus to increase density and efficiency. The platform OpenShift (Kubernetes) allows users to control the scaling of the instance. Since it’s on K8s, another advantage is that operation can operate these as a unify platform, along with all other microservices.

Why Kamelet? (This is the way!)

One of the biggest hurdles for non Camel developers is, they need to learn another framework, maybe another language (Non-Java) to be able to get Camel running. What if we can smooth the learning curve and make it simple for newcomers? We see a great number of use cases where the masking and filtering are implemented company wide. Being able to build a repository and reuse these logics will make developers work more efficiently.

Plug & Play

You can look at Kamelets as templates, where you can define where to consume data from and send data to, does filtering, masking, simple calculation logic. Once the template is defined, it can be made available to the teams, that simply plugs it into the platform, configure for their needs (with either Kamelet Binding or another Camel route), and boom. The underlying Camel K will do the hard work for you, compile, build, package and deploy. You have a smart running data pipeline streams into Kafka.

Assemble & Reuse

In a data pipeline, sometimes, you just need that bit of extra work on the data. Instead of defining a single template for each case, you can also break it down into smaller tasks. And assemble these small tasks to perform in the pipeline for each use case. .

Streams & Serverless

Kamelets allows you to stream data to/from either Kafka store or Knative event channel/broker. To be able to support Knative, Kamelet can help translate messages to CloudEvents, which is the CNCF standard event format for serverless. And also apply any pre/post-processing of the content in the pipeline.

Scalable & Flexible

Kamelet lives on Kubernetes(can also run standalone), which gives you a comprehensive set of scaling tools, readiness, liveness check and scaling configuration. They are all part of the package. It scales by adding more instances. The UI on the OpenShift Developer Console can assist you to fill in what’s needed. And also auto discover the available source/sink for you to choose where the data pipelines start or end.

Unify for DEV & OPS

In many cases, DevOps engineers are often required to develop another set of automation tools for the deployment of connectors. Kamelet can run like other applications on kubernetes, the same tools can be used to build, deploy and monitor these pipelines. The streamline DEVOPS experience can help speed up the automation setup time.

Marketplace

List of catalogues that are already available (Not enough?). If you just want to stream data directly, simple pick the ones you need and start streaming. And we welcome your contributions too.

What to know more about Kamelet? Take a look at this video, where it talks about why using Kamelet for streaming data to Kafka with a demo.

00:00 Introduction

00:30 What is Kamelet?

01:06 Why do you need connectors to Kafka, and what is required in each connector?

02:16 Why Kamelet?

07:45 Marketplace of Kamelet!

08:47 Using Kamelet as a Kafka user

10:58 Building a Kamelet

13:25 Running Kamelet on Kubernetes

15:44 Demo

17:30 Red Hat OpenShift Streams in action

19:26 Kamelets in action

Tooling guide for Getting Started with Apache Camel in 2021

2021-05-26T21:20:00.002+08:00

Getting Started with Apache Camel ? This post is for you. But I am not going to dive into how to write the Camel route, there are plenty of materials out there that do a better job than me. A good place to get started is the one and only Camel Blog, it gives you all the latest and greatest news from the community. If you want to start from the basics, I highly recommend Camel in Action II Book. It has everything you need to know about writing Camel. Or join the Camel community mailing list or Zulip chat to ask questions, it's very friendly and welcoming.

If you are a returning Camel rider, I would go through this post, where it will get you up to speed on what to expect. Another good resource is the FAQ page. I found the majority of the getting started enquiries can be found here.

This post will be about tools that can help you, when it comes to running a Camel application in its lifetime. From design, implementation, testing, deployment and monitoring. Of course this is strictly my opinion. Any suggestions or comments are welcome.

Here is a quick collection of the tools I recommend.

Design & Implementation

There are several different IDEs out there that have language support for Camel. My current goto is the Visual Code Studio. VS Code itself has very primitive support for Java developers, but it comes with a large variety of extensions for you to customize it, and install all the tools you need. Speaking of extension, there is a must have… “Extension Pack for Apache Camel by Red Hat”. It contains the essentials for developing Camel. I recommend checking out the Aurélien Pupier youtube channel for updates!

To begin a project, it’s always good to have some help to set the basic structure. You can either use the Project Initializer that comes with the Apache Camel by Red Hat Extension in VS Code that will generate the base project for you. Or you can always go to the Quarkus start coding page, adding the Camel dependencies to create the base project, but one downside of the Quarkus start coding page, it will not create the template extended RouteBuilder class for you.

No matter what runtime you choose, you are still writing Camel DSL (Domain Specific Language), you have the latest and greatest support from the camel extension, where it will help you with autocomplete, validating your Camel code.

Mapping data between formats can be tedious, especially with larger documents or more complex structure. But using the

tooling can help you map between two data sources with a drag and drop user interface, so you will be able to visualize the mappings. The tool will generate an “.adm” file, place the file into your Camel project. You can then use the camel-atlasmap components to run the mapping.

from("servicenow:xxxxxx....")

.split().body()

.marshal().json()

.to("atlasmap:servicenow.adm")

.to("kafka:xxx?brokersxxx")

RESTful API is another popular implementation in Camel, I highly recommend you take advantage of the Apicurio project, where it provides a GUI interface for designing the API contract. Making contract first application development a lot easier. For more details, take a look at my previous blog.

Another nice complements to the toolset is the Camel Designer, this is another extension in VS Code by Bruno, this can help you visualize the design of camel processing flow. So you will be able to see what your camel route does in a quick glance.

Testing

Camel can easily wire up unit tests in your preferred testing framework. My past experience has been great with using JUnit for creating my test cases.

What I normally would do is create a java Class for each test case (Extend “CamelTestSupport” class). Where it can kick off your camel route with mock endpoints, loading set of data or even stub out the actual endpoint

When testing integration, introducing Behavior-Driven Development(BDD) Testing is a development practice, as it’s black box testing nature allows it to better mimic user input and expectation, and can test more complex scenarios when compared to Test-Driven Development(TDD). Consider using the “Citrus Testing Framework” for building out BDD test cases. You can define the test scenario descriptions with a "Given-When-Then" structure in a feature file, as well as a simple setup for mimicking the common endpoints such as Kafka, AMQP, JMS, REST… etc.

Feature: integration runs

Background:

Given load variables namespace.properties

Given URL: http://camel-hello-quarkus.${namespace}.svc.cluster.local:8080

Scenario: Checking GET Method

When send GET /prescription?profileid=123456

Then verify HTTP response body: {"profileid": 123456, "pharmancy": "CVS" }

Then receive HTTP 200 OK

I have also been using a spinoff project from Citrus called YAKs to do my BDD testing on Kubernetes(OpenShift) platform. And I LOVED it. Simply because 1st, the lifecycle of the test was managed by the YAKs operator, meaning the test was run on a separate instance, that is where the client supposed the call, and also the YAKs operator takes in the feature, and does the test based on it. You can run it separately or plug it into your CI/CD pipeline.

Platform

I have been using OpenShift(Kunerenets) for managing all the running instances. It's just easier when there are hundreds of Camels running, it manages the scaling, resources, configuration and load-balancing for me. As well as a better deployment module (images). There are so many articles out there talking about the benefits of running a container & container management, so I won’t dive into it.

CI/CD

Jenkin was on my list for a long time when it comes to building the pipeline for my Camel application. Basically using the maven jenkins plugin for compiling and deploying, and using the OpenShift plugin to do the canary releases. And recently, I have been using the Tekton pipeline, it’s built-in on OpenShift, so there is no need to install a separate instance and manage it. The Kubernetes platform itself will become the CI/CD platform itself. Here is an example of how it works with the Camel projects. I will write another deep dive into how it works.

Monitoring

With Camel 3, in order to collect and export the metrics, you will need to specifically add the microprofile-metrics dependency. And by using Prometheus to scrape metrics on the platform. It stores all scraped samples locally and aggregate the time series data. I then use Grafana to visualize the collected data, and create a dashboard to monitor all the camel routes.

Also checkout this video and see how things work:

Contract first development - the event driven way!

2020-11-09T23:53:00.001+08:00

Introduction:

Contract first application development is not limited to synchronized RESTFul API calls. With the adoption of event driven architecture, more developers are demanding a way to set up contracts between these asynchronous event publishers and consumers.. Sharing what data format that each subscriber is consuming, what data format is used from the event publisher, in a way OpenAPI standards is going to be very helpful.

But in the asynchronous world, it is ever more complex, not only do you need to be caring about the data schema, there are different protocols, serializing, deserializing mechanisms, and various support libraries. In fact there are talks on AsyncAPI. But I am going to show you that what can be done today, is to use ONE of the most commonly used solutions in building EDA, “Kafka”. And show how to do Contract First Application Development using Camel + Apicurio Registry.

The Solutions:

I will be using Red Hat’s version of Kafka, AMQ Streams, APIcurio as my service registry and finally using Apache Camel to build the contract first event driven application. . To spice things up I am going to be Avro and Protobuf. (Normally you want to choose between one of them.)

Rather than sending plain text like JSON or XML, binary representation is more compacted and efficient. And this works well with Kafka, since Kafka is using binary message format. In short, Avro and Protobuf are just ways to serialize/deserialize the data. Both also provide schema in order to serialize/deserialize data. And we can use it as the contract for each topic that is publishing/consuming the events.

Avro is from Hadoop, more commonly used in Kafka, as it’s default serialization.

Protobuf is from Google, has two versions, here I am using proto 3.

Apricurio Registry is a datastore for standard event schemas and API designs. We are going to use it for our Schema management. Where we will use it to store all the schemas, and use it’s client serializer/deserializer to validate the event messages.

Camel is the perfect framework for connecting the dots, transforming messages to desired states. As it provides built-in libraries to connect to Kafka, built-in data format transforms for Avro/Protobuf. (This is a great feature if you want to process the body, it simplifies and marshal the data into POJO.)

There are two different approaches to do this:

Manually upload schema, and using default Camel dataformat component to serialize/deserialize
Using the Apicurio Registry libraries to upload schema and serialize/deserialize

Environment

Here is what you need to have in the environment.

Kafka Cluster (AMQ Streams)
Apricurio Service Registry

Solution A Camel built-in Dataformat components: Steps in a glance, using Camel Quarkus.

Step One:

Go to this page to https://code.quarkus.io/ to generate your bootstrap quarkus application.

You will have a sample Camel project ready to go. Add all the dependencies needed for the project. Depending on the endpoint you want to connect to, and add the dependencies into the pom.xml file under the project.

Add the dependencies you will need to convert between two serialize mechanisms, and place it into the Kafka topics.

<groupId>org.apache.camel</groupId>

<artifactId>camel-quarkus-protobuf</artifactId>

</dependency>

<groupId>org.apache.camel</groupId>

<artifactId>camel-quarkus-avro</artifactId>

</dependency>

<groupId>org.apache.camel</groupId>

<artifactId>camel-quarkus-kafka</artifactId>

</dependency>

<groupId>org.apache.camel</groupId>

<artifactId>camel-quarkus-stream</artifactId>

</dependency>

Add plugins to download the schema from registry. (Of course you can register a schema from local code too). But I am assuming there is a separate person that will design and handle schema management.

<groupId>io.apicurio</groupId>

<artifactId>apicurio-registry-maven-plugin</artifactId>

<version>${registry.version}</version>

<phase>generate-sources</phase>

<goals>

<goal>download</goal>

</goals>

<registryUrl>http://localhost:8080/api</registryUrl>

<ids>

<param1>demo-protobuf</param1>

</ids>

<outputDirectory>${project.basedir}/src/main/resources</outputDirectory>

</configuration>

</execution>

</executions>

</plugin>

Note: In configuration, enter the location of the Registry. Also make sure you tell the plugin what schema to download and where to place it in the project.

Step Two

Create the schema. Login to your Apricurio Service Registry, upload the schema.

Setup the topics that you want to use to populate the events.

Add plugins to generate the POJO that we are going to be using for marshalling/unmarshalling. And it’s easier if we want to further process the data.

<groupId>org.apache.avro</groupId>

<artifactId>avro-maven-plugin</artifactId>

<phase>generate-sources</phase>

<goals>

<goal>schema</goal>

</goals>

<sourceDirectory>${project.basedir}/src/main/resources</sourceDirectory>

<outputDirectory>${project.basedir}/src/main/java</outputDirectory>

</configuration>

</execution>

</executions>

</plugin>

<groupId>org.xolstice.maven.plugins</groupId>

<artifactId>protobuf-maven-plugin</artifactId>

<goals>

<goal>compile</goal>

</goals>

<protocArtifact>com.google.protobuf:protoc:${protobuf-version}:exe:${os.detected.classifier}</protocArtifact>

<outputDirectory>${project.basedir}/src/main/java</outputDirectory>

<protoSourceRoot>${project.basedir}/src/main/resources</protoSourceRoot>

<clearOutputDirectory>false</clearOutputDirectory>

</configuration>

</execution>

</executions>

</plugin>

After adding the plugins, run the following command:

`` mvn compile ``

It will generate the corresponding java source base on the schema that was loaded from the registry.

Step Three

Add your Camel route in the MyRouteBuilder.java. Depending on what you want to do with the object.

For example, the code here demonstrates how to use the built-in Protobuf component, and transform to the Protobuf object. Before sending the data as Byte Array into the Kafka Topics.

.marshal().protobuf("demo.camel.TransactionProtos$Transaction")

.log("Sender: ${header.sender}") .toD("kafka:webtrans-quarkus?brokers=localhost:9092&key=${header.sender}&serializerClass=org.apache.kafka.common.serialization.ByteArraySerializer")

Or this shows you how to use the built-in Camel Avro component to transform before sending the data as Byte Array into the Kafka Topics

AvroDataFormat format = new AvroDataFormat(Transaction.SCHEMA$);

....

.marshal(format)

.toD("kafka:transrec-quarkus?brokers=localhost:9092&groupId=producergroup&serializerClass=org.apache.kafka.common.serialization.ByteArraySerializer")

And it’s done.

Here are the working Camel Quarkus Project:

I will discuss more in the next post.

-----

Instead of manually uploading the schemas to the registry, we will be using the Apicurio Registry libraries, by adding it to the Kafka configuration, it will automatically save the schema in the registry.

Here is how it’s done,

Solution B - Apricurio Registry Libraries : Steps in a glance, using Camel Java Main.

Step One:

Create a Camel Project using the archetypes.

mvn archetype:generate \

-DarchetypeGroupId=org.apache.camel.archetypes \

-DarchetypeArtifactId=camel-archetype-java \

-DarchetypeVersion=3.5.0

Add the dependencies you will need to convert between two serialize mechanisms, and place it into the Kafka topics.

<groupId>io.apicurio</groupId>

<artifactId>apicurio-registry-utils-converter</artifactId>

<version>${registry.version}</version>

</dependency>

<groupId>org.glassfish.jersey.core</groupId>

<artifactId>jersey-common</artifactId>

</dependency>

<groupId>org.apache.camel</groupId>

<artifactId>camel-kafka</artifactId>

</dependency>

<groupId>org.apache.camel</groupId>

<artifactId>camel-stream</artifactId>

</dependency>

Step Two:

Create the schema and place the schema files into your application resource folder:

${project.basedir}/src/main/resources

Setup the topics that you want to use to populate the events.

Add plugins to generate the POJO that we are going to be using for marshalling/unmarshalling. And it’s easier if we want to further process the data.

<groupId>org.apache.avro</groupId>

<artifactId>avro-maven-plugin</artifactId>

<phase>generate-sources</phase>

<goals>

<goal>schema</goal>

</goals>

<sourceDirectory>${project.basedir}/src/main/resources</sourceDirectory>

<outputDirectory>${project.basedir}/src/main/java</outputDirectory>

</configuration>

</execution>

</executions>

</plugin>

<groupId>org.xolstice.maven.plugins</groupId>

<artifactId>protobuf-maven-plugin</artifactId>

<goals>

<goal>compile</goal>

</goals>

<protocArtifact>com.google.protobuf:protoc:${protobuf-version}:exe:${os.detected.classifier}</protocArtifact>

<outputDirectory>${project.basedir}/src/main/java</outputDirectory>

<protoSourceRoot>${project.basedir}/src/main/resources</protoSourceRoot>

<clearOutputDirectory>false</clearOutputDirectory>

</configuration>

</execution>

</executions>

</plugin>

After adding the plugins, run the following command:

`` mvn compile ``

Step Three:

Add your Camel route in the MyRouteBuilder.java. Configure the Kafka component with the Apicurio registry libraries.

For example, the code here demonstrates how to serialize Avro and deserialize Protobuf with Apicurio registry libraries. And automatically register the schema in the registry. Note you need to point to the registry URL by providing the “apicurio.registry.url”.

serializerClass=io.apicurio.registry.utils.serde.AvroKafkaSerializer

deserializerClass=io.apicurio.registry.utils.serde.ProtobufKafkaDeserializer

additionalProperties.apicurio.registry.url=http://localhost:8080/api

additionalProperties.apicurio.registry.global-id=io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy

And in the Camel route, instantiate the two generated Avro/Protobuf classes aldds, work normally with your camel integration logic, and simply send it to the Kafka endpoint that was configured on the previous step.

For example, this bean will create the ProtoBuf class and then sent to Kafka topics

.bean(demo.camel.TransactionProtos.Transaction.class, "parseFrom(${body})")

.toD("kafka:webtrans?key=${header.sender}")

For example, the method will create the Avro class and then sent to Kafka topics

.setBody(method(this,"setAvroValue( ${originalmsg.userid},${originalmsg.amt})"))

.toD("kafka:transrec?serializerClass={{serializerClass}}"+registryconfigAvro)

You can find a working github example, in the link.That’s all for this post.

In my next post, I will walk you through the details of the example.

Summary

Here is a video that explain everything with diagrams (NO CODE)

Apricurio Registry is a datastore to help topic owners to manage the schema of it’s dat format. (API for Rest endpoints). When trying to publish/subscribe to the topic, developers will have access to the already defined schema contract. And using Apache Camel, to implement the actual integration code with simple configuration or built-in data format components.

Serverless Integration Components

2020-07-09T02:16:00.000+08:00

Serverless should not be optional, but instead it should be there for all cloud native environments. Of course not all applications should adopt serverless. But if you look closer, the majority of the modules in the application are stateless, often stash away in the corner, that are needed occasionally. Some need to to handle loads that are highly fluctuated. These are the perfect candidates to run as serverless.

Serverless let developers focus on code instead of worrying about infrastructural setups. To provide this environment, along with proper monitoring and reliable backbone to handle large throughput of events.

This is what Serverless Integration (Kubernetes based) looks like,

Everything is based on containers. Why care about the underlying technologies for serverless ? Shouldn’t it all be transparent? If your goal is to host or build on a hybrid/multi cloud that is locked-in free from all vendors, it means there are NOT just developers involved in the picture. You will eventually need cooperation between teams and work with all sorts of applications such as traditional services and microservices. Having unification and standardization technology, will flatten the learning curve for teams to adopt new kinds of applications and make maintenance less complex.

From the development to the platform everything should seamlessly work together, and being able to automate and manage with ease.

Let’s break down all the elements.

The Platform: A Platform that provides full infrastructure and platform management with self service capability, service discovery and applying container policy and compliance.

The Serverless Platform: Handles autoscaling of the functions/application. Abstraction of the underlying infrastructure. Setup revision of the deployments for easy rollback. And unify events for the publishers and consumers.

The Event Mesh: Events are published to the mesh, and to the distributed consumers. The basic structure of the events are consistent and should be portable among platforms. All the events are flexible, governed and pushed quickly. Powered by a reliable streaming network, it helps to store the streams of event for tracing, auditing or later replay for big data processing/ AI/ML datasets.

The Integration Functions: Typical characteristics of serverless integration include, small, lightweight, stateless and event driven. These characteristics allow the application to be elastic, to tackle the under/over provisioning that we face today. From the operation side these are the applications that cease and quickly spin up after being triggered by events. For better resource optimization. And for developers it is a simple modular code snippet that they write and gets automatically spun up. So they can focus on code instead of deployment related issues. And Integration functions are the application that typically handles routing, transformation of data payload in events, and also other composing and orchestrating problems. Also commonly used for connecting to external services, and bridges between systems.

The microservice or the long running applications: These are the long running applications that contain states, heavier or that is always being called. Some of them will send events to the mesh to trigger serverless functions to initiate and start, or simply another consumer of the events.

The Service Registry: For sharing standard event schemas and API designs across API and event-driven architectures, either for events listening by serverless function or regular applications. Decoupling data structure and managing data types at runtime.

The API management: Gateway to secure and manage outgoing API endpoints. With access control and limits for the consumer, managing consoles and data analytics for access endpoints.

These are again, my two cents on the components that you need to have in order to deliver a complete serverless application environment.

Six reasons why you will love Camel K

2020-04-23T20:30:00.000+08:00

Camel K, a project under the famous Apache Camel project. This project totally change the way developers work with Kubernetes/OpenShift cloud platforms. By automating the nasty configuration and loads of prep work from developers.

If you are an old time developer like me. I did my best slowly trying to adapt the latest and greatest cloud native “ecology”. It’s not difficult, but with small things and traps here and there. I’ll tell yel’ it's not a smooth ride. It’s understandable for emerging technologies. But with the large adoption of cloud, I see it’s reaching a level of maturity, where now we are thinking of how to make things go faster, as well as making it more accessible to the larger audience.

And here is why you will love Camel K..

Real-time coding on the platform

When a developer wants to publish their application onto a Kubernetes/OpenShift Platform, they need to first build the app. And containerized the app to have a working image. Then pushing the image to the platform to run. If the developer is trying to debug or develop the app with some changes, sorry, go back to square one and build it again. The worst nightmare for developers is the wait. It’s time consuming, and breaks the chain of thoughts. Camel K eliminates the painstaking process. Let developers quickly update apps they are building on the platform immediately. By layering the image build on cloud, and streaming the code efficiently with the Camel K operator.

Natively on cloud

Camel K is smart to discover the surrounding platform resources required to run on the platform. For instance, if a developer exposes a HTTP endpoint in the app, Camel K will know to create a related service and route in the platform. Another example is the cron jobs, where Camel K configures it for developers when it sees the behaviour in the code. Configurations are also taken care of for developers with Camel K convert the properties files into Kubernetes resources too. Reversing it is also simple. Simply remove the app, and Camel K will delete the related resource.

No more dependency hell

Dependency is troublesome, annoying things to a developer. It’s not a big deal in the monolithic world, it probably imported everything in the world already from prior projects. (Let’s not talk about the redundancy, wasted resource and version conflicts). When writing a 20 lines small microservice or function, it doesn’t make sense for a developer to write a 50 line dependency definition. Camel K will smartly pick up the dependencies needed to run the app, it will automatically locate the libraries when building. But also give developers the choice to load any specific libraries they need as well.

Freedom to choose the language and runtime

Camel K supports range of languages for developers, no matter if it’s Java or scripting languages like Groovy or Javascript to markup languages like XML or YAML. Camel K no longer needs a big fat framework like Springboot to run. Depending on the types of application, developers can choose to run simple plain small Java Main in their microservice. Or go with the latest and greatest Quarkus with fast boot up time for serverless apps. Running the runtime requires literally just ONE configuration. “-t quarkus.enabled=true”

Simple, Small but Powerful

If we look closely at Camel K app, the core is from Apache Camel. This means developers will have well established patterns to reuse. Pre-built transformation for data and 300+ components available for quick connectivity. No more reinventing the wheel. So the code is cleaner, easier to maintain, but also maintains highly customizable freedom for users.

Serverless made easy

Reduce the operation, development and scaling cost. KNative is the most popular serverless frameworks on Kubernetes. It introduces the idea of scaling to zero, scaling up for load. And also serving the new specification for future Cloud Events. Applying these serverless capabilities to an app on the cloud requires a set of configuration. But with Camel K it will automatically do it and make it seamless for developer trying to access the channel to get the cloud events.

Check out this video I made a couple of months ago on Camel K. A 1000 feet high overview of what Camel K is. And also the getting started series I made for Camel K.

Lastly, choose the right tools for your projects. Camel K may not be a silver bullet that magically solves all your problems. But it’s definitely worth your time to see how dramatically it is going to change the way you code on Cloud platform!

My2Cents - Eight things leads to developing catastrophic cloud native microservices system

2019-03-19T22:21:00.002+08:00

More of my two cents, just my thoughts. A quick fun read, not too deep, but worth noting :).

1. Setting the domain boundary wrong

This is a job guarantee tactics, it's endless looping in development and testing for everyone involved in the project without making the service to production! First everything starts simple and gradually find more and more functions, business logic gets added into the microservice, at the end, one even have to rename the whole damn thing.

Symptoms and side effects

A growing microservices becomes too fat, or every single microservices in the domain calls your microservice. (Sometime core microservice has the same behavior, but you should not see many of this type of services in a single domain.) This violates the microservices principle of easy, maintainable and agile.
Duplication of microservices/code everywhere. Where you find bits and pieces of duplicate code or microservices being copied and deployed into other domains.

If you get into the endless implementation and testing hell, take a step back and look at how you separate the domains. Did you acc
identally place a context from other domains or mixing different concepts int
o one? Maybe it's worthwhile to go back to the design phase and think about the boundaries. To avoid duplication everywhere, make sure there are proper documented catalogs such as API portals using the Open API Standard Doc available between domains.

2. Mixing responsibilities of microservices

In the mood for some Spaghetti, this is the way to go, and want some meatball to go on the Spaghetti to make it tastier? Mixing some stateful processes in there will get you plenty!

Mixing

Composing microservices
Functional core business
Stateless integration
Stateful process

Symptoms and side effects

Client calls becomes complex, if you don't provide a composed view of particular service, the client may require to compose and possibly handle some sort of core business login in it.
Spaghetti relations between services, and it's really easy to lose track of how the flow of data and business logic spread around microservices.
Too much coupling, having stateless and stateful is depended on the storage solution and scaling down could also have potential problems.

I've done a bunch of blogs on this topic, the main concept is always the separation of concerns, making sure there your cloud native microservices are clean, lean and agile meaning you need to make sure the business composition and the actual business functions needs to be separated into different instance and deployed individually for better structured and more modularized. Making sure the composition/orchestrations of microservices are stateless for scalability and flexibility. Set aside other long running process on other instance with dedicated proper storage solution to keep the states.

3. Too much coupling with external factors/demands

Being patient is always good, if you want to practice your ability to be extra patient while being calm, by coupling deeply with legacy system will train you to become the patient master, and become better at code management as you need juggle between agile demands and slow legacy update cycles.

Symptoms and side effects

Constant updates to core business to the core business system outside of better control. Having external system decide and make change request and short reaction time could cause issues.
Once contract are establish between parties, will need extra care for version updates.

The deployment cycle varies from system to system, but brown field application will have a longer application lifecycle compare to more agile and dynamic cloud native application. To move things faster, always consider to have a shield protection between your brown/green field application. So it accommodates to the s
lower moving parts as well as help green field cycle to continue on it's agile development. In your core domain microservices, avoid add anything that is dependent or specific for external consumers, as it might leads to too much customization that breaks the definite context boundary. Make sure you have other vendor/external system facing module that does the customization and hide the presentation complexity from internal domain models.

4. Duplicating and replicate non-biz related code in microservices

As a developer, it's nice to be depended on, you feel the satisfaction of being needed. There is a reason why there are 24 hours each day, so you get to work 20 hours per day and you are so important when ever there is a deployment, you needed to be there.

Symptoms and side effects

None centralized control, all retry, reroute, versioning and deployment strategy are store in each individual instance.
Responsibility not clear between DEV/DEVOPS, again, the networking strategy is better for party that has better access to monitoring stats of how the environment is doing.

Setting retry and routing policy can allows application to be more robust to failure. To have a more centralized view of how these policies are handled instead of spreading them all over individual microservices. Release the burden of developer handling everything in the application and better management and monitoring.

5. Service in mesh, connecting all with APIs

Making API calls are super easy, and service mesh is what everyone is doing today. Every technology has an library to handle REST/JSON. Let's connect all components in the system with APIs!

Symptoms and side effects

Slow to react, since request and response are how most people uses it. And the wait could take longer and more locks for longer process. That becomes the performance bottleneck.
Not utilizing cloud infrastructure.

When microservice first came out, people wants to get away from ESB therefore many moved away from messaging broker and use API as the only connection method between services, as the Open API standard allows you to build a nice catalog among what is available. But event driven is there for a reason, to achieve true scalability, better decoupling, you need to stop making sticky dependency between calls. And event driven allows you to populate events to related parties, and because it's asynchronies, no resource are wasted when waiting for the reply.

6. Un-sorted events

Blasting the system with events, notify every corner to any events, will get your system super reactive! By the way, why not stream everything and store all the event! For better traceability!

Symptoms and side effects

Events everywhere, you realized you have to listen to insane amount of events to react up on it.
Un-needed code to filter out events.
Confusion between status change or action should taken
Don't know what is better to preserve or lose the event messages

There are various types of events, if not carefully where and how the events are distributed, it's very likely to end up with unwanted events floating around consuming unnecessary resources. Focusing on some of the common ones like events containing data, status and commands. You want to minimize the number of data messages, as it take longer to process, more work to store. Striping down, filter the data could be more efficient. Handling retries of status events and rolling back command events should also be part of the event strategies.

7. Data Silos

One mircroservice one datasource, that is what experts says.

Symptoms and side effects

Data not consistent among the datasources, or slow.
Un-needed microservices created just for making sure the data are consistent.
Events everywhere to update status.

When you have isolated datasources there are risk of creating data silos, and because of the nature in microservice being distribute and more complex datastore scenario, it has become more difficult to mange the data consistency. You can start looking into how to capture the data change with many existing solutions such as streaming event changes as it happens during process or listen to changes from the main status store.

8. Limited automation

Manual is much more flexible, beside that was how it was done before with the JavaEE application on the servers. Automation, it takes too much time to setup, Lets deal with it when we have time?

Symptoms and side effects

Slow updates
Long painful steps to productions

The first thing that needs to be in the planning is definitely automation. How to apply automated CI/CD pipelines and the deployment strategy, this how cloud native microservices system achieve the agility. I've done a couple of CI/CD automation with Jenkins in the past. See Github Link.

Alright, these are just my two cents, let me know if you think there are any other things that could lead to terrible cloud native microservices development? Love to know!

AMQ Online 101 - What I've learnt from recent AMQ Online Hackfest

2019-01-29T22:15:00.000+08:00

AMQ Online, project name, EnMasse. Recently, I've attended AMQ Online Hackfest where the Rob and Ulf the engineering team kindly ask us to take a hack on the new platform. If you don't know what AMQ Online is, it's basically an online self-service (Messaging as a Service) messaging platform for developers/application owners to quickly spin up their own queue and topic without worrying about the scalability, network setting and all sorts of operation setups.

Watch this quick video overview on what is AMQ Online:

Before the hack, they went through the basics of AMQ Online, and here is my take on explaining the basics of AMQ Online. With AMQ Online being the platform to service users, there are obviously two main area of focus for people depending on their roles. The administration side to spin up and manage the platform, the other side of course is the user end, where they will be requesting for the message services.

Admin does the installation of the platform foundation on OpenShift, they install all the basic elements that controls all the needed brokers, routers, user console and monitoring tools, and also takes care of setting up the authentication and authorization of who has access to the platform. We don't always have unlimited computing resource for all users, admin also has control over the resource limitation and create different plans that can apply to accompany different needs from users.

Each user will be assign to an "Address Space", each Address Space will have separate sets of brokers, router and console for it, and later on when user comes in to ask for queue and topics, they will be running on top of these set of brokers and routers. Admins does not create the actual "Address Space", their job is to create PLANs that specify the limits, configuration. So when a new user comes in, they can then create the "Address Space" base on the plans.

There are several plans that can be defined.

Infrastructure configuration - as name suggested, it configures the hardware resource allocated to the components, such as CPU. memory, storage of console, broker and router, also some other specific config too.
Address Space Plan - basically the main quota control, it knows how much hardware can be used by referring to infrastructure configuration, as well as controls the number of routers/brokers in the system. And so what are the queue/topic plans are available in the address space.
Address Plan - Address plan defines the consumption of resource base on the percentage. The sum of number of brokers/routers of the entire address space will determine how many instance of broker and router will be running in the name space.

Tenants are the users of the platform, they will be creating "Address Space", and create destinations by applying the plans on top of it. After the queue and topics are created, the tenants can go ahead and create credentials maps to different access roles, they can decide what account has access to admin console, which account has consume/produce right to any addresses(queues/ topics) in the Address Space. Please note, tenants will have no visibility to queue or topics of other Address Space, unless specially specified. Once the address are created and config correctly, application will be able to use the credential to read/write to the messaging address.

These are the three things tenants need to create

Address Space - refer back to Address Space Plan and create and config pod base on the plan selected.
Address - the destination create for user access. Maps to address plan.
Message User - defines the authentication method for destination access, and the authorization of destinations to this user.

Watch this video on AMQ Online 101:

There are different types of Address Space, Address.. and other configs available. I want to keep this post as simple and basic as possible, please read into the documentation for more details. I might go over in next post, if I ever get some spare time to do it :) Till next time! Enjoy.

My 2cents on the future of Integration - With Service Mesh/Istio and Serverless/KNative

2019-01-22T10:32:00.000+08:00

It's been a year and half since I blogged about "Agile Integration architecture" (Gosh, time just flies). With the "microservices" and "cloud-native" hype, I was especially curious on how all these new concept and technology affect us on how to architect the integration systems. If you ever pay close attention to all the latest and greatest news from the Kubernetes community, I am sure you will hear a lot about the new "Service Mesh". And rumor has it that this is how integration can/should be done in cloud native world, but, is that so? Anyone who has ever worked on an integration project would tell you, it's a LOT more COMPLEX and can get worst overtime. I did a talk with Christian Posta in Red Hat Tech Exchange coming from a more comprehensive view of how different Red Hat technologies are applied under different patterns when building integration solutions. In fact he also did a great blog about it.

Since then, another topics has been brought up numerous time, what about Serverless, how would it impact integration? Does it mean the death of services? If we have all Microservices connected via Serviceless mechanism in service mesh, does that mean we go back to the old days of writing all the integration transformation logics in our application again? Obviously, this is complicated question to answer, and the new features keeps popping up, I am going to try to give my best to try and explain how I see everything fits into Agile Integration vision. Honestly, there is no exact right answer when it's about architecting system, it requires constant refactoring, my thought is to give a more general and flexible way of doing it that has less impact when change and being able to adapt change quickly. (That is what I call Agile Integration).

Service Mesh

I remember last time, I started the agile integration reference architecture simply because the lack of organization of Microservices, and it still stand today. JUST because we have service mesh, doesn't mean it will magically solve the spaghetti connectivity if you are not being careful. To set the ground, what service mesh helps is to relief developer from dealing with repetitive, boring work in a distributed environment. Yes. Being cloud native with Microservices, means the system is more vulnerable to chain reaction disaster if not designed correctly, as you are working with small bots and gears in the system, that you never know how any small missed-handle can impact the entire system. Therefore we need to make each component more robust, failure proof and ideally damage recoverable. And Service Mesh helps you with getting the BASICs ready by adding a sidecar next to your running application. For every Microservices you create, no matter what they do, core business or content orchestration, they all need some kind of failure proof. Service Mesh gives you all that, from an application networking level. So that as a developer you will no longer have to wrap every single one of your Microservices with the circuit breaker, error retry and even handling version, deployment routing, and apply authorization, those that has NOTHING to do with what the actually app is responsible for. These are often common rules that applies to the whole cluster. And service mesh is best for that, as these common policies are detached from actual application, can be managed centrally and apply individual on the sidecar so the microservice running your app is protected behind it.

Another great feature with service mesh, is to ability to trace and observe the incoming request for the entire system. This will certainly help developer/operation have more insight into the complex spider web-like connectivity.

Dealing with REAL Life Integration
But when it comes to actually application implementation, some folks thinks we can now rely on Service Mesh to configure and connect. MAYBE, if you are developing only a couple of small microservices, you can probably get away from that. BUT that is not what's happening in real life. For instance, formatting the data to the right granularity (Splitting/Aggregation), routing base on processing outcome of some content, a more complex orchestration of services call that requires precise rollback and business handling (Saga) , collection of events triggering, we can't possibly write all these into the Service Mesh config yml file (I am just being practical :p). So, I am sure now we have mutual agreement that integration logic still needs be written somewhere, DON'T tried to do it in your microservice with the business logic. That is why we need the conceptual layer in agile integration. They are the composite and core layer. Remember, I mean CONCEPTUAL not physically on top of another, but separating the responsibility of the microservices so it's easier to locate, maintain and organize your applications. Taken from the lesson learnt from SOA. You will still need some kind of integration patterns to compose service that has the right granularity for the receiving end.

Monitoring and tracing is crucial in ANY integration system, and let's face it, there are NOT JUST http calls. Majority of times, they are events, and honestly an event driven reactive system is 100 times more flexible and modularize than sticking a bunch of API together. To be able to collect these data are needed beyond simple request tracing.

Serverless
I had to talk about this, the whole serverless development, deployment and runtime concept can take agile integration in another level not just conceptually but physically too. The idea of quickly spin up integration piece of logic, the ability to scale freely up and down. That is what I call true agility. I was thrilled to see Nicola demonstrate Camel-K on KNative. Camel just opens up a wide range of connect possibility for the serverless call. To me the core spirits of server less is (For now)

Being able to quickly produce/start application without complex configuration and heavy runtime.
Elastic resource allocation responding to loads.

And what integration helps to bring to the table is the ability to allow the system to responds more events from boarder range of endpoints, for instance, IoT, SOAP and messaging and other protocol that are not simple http calls.

This is just a HIGH level overview of how I see Integration are shaped in the future of a serverless cloud native world. Of course I have not touch upon many topics like APIs, Event driven architecture, self service ability. They deserve separate blogs. Might do another later this week. Again, these are just my 2cents coming out from a more practical real life point of view.

Fuse - Contract First API Design with Apicurio and Fuse/Camel - Part One

2018-07-06T01:46:00.000+08:00

This is part one of my two-article series that demonstrates the approach of implementing contract-first API design using Apicurioand Red Hat Fuse.

It covers how to create an OpenAPI standard document as the contract between API providers and consumers using Apicurio. It also shows how to quickly create mock tests using Red Hat Fuse.

There are two common approaches of creating these APIs.

Code First
Contract First

Coming from a old time ESB developer, these are not new. We have been doing this forever. Before, it was the WSDL that define the contract of the service. we were doing a lot more code first, for me it's simply because it's much easier for me to write couple of Java classes and generate the WSDL for my consumer.

It's often pretty straightforward if the consumer of your application has finalized how they want the service to be like. But you and I all know this is not often the case. So I had to go back to my code, and make the changes accordingly and pray I did not break anything in the way.

Of course there is always preliminary discussion, but small changes are inevitable, and it's taking the toll of developer to grind through the process of getting everything right.

Thanks to the new OpenAPI Specification, business user, citizen user/developer can use it to negotiate and maybe rung a couple of pre-test with the consumer before it gets handed over to developer. This design approach is called Contract First. It has become more and more popular as it avoids developer wasting time while in the process of negotiating how the service should be provided.

Obviously there are many ways to do Contract First API . I am going to demonstrate how this can be done using Apicurio and Fuse. I will be using Apicurio to define APIs, and automatically generates the Fuse project for quick testing purpose.

Theme: Customer Service API

We will be providing customer service to our consumer. For the sake of this demo, we will only start with retrieving and creating customer service.

Defining Application with OpenAPI Specification in Apicurio

Apicurio is a web-based Open Source tool to design OpenAPI specification. Go to https://www.apicur.io/

If you don't already have an account, please register for one.

After registering, you will be redirected to the main page of Apicurio, so after discussing with the consumer on what they wish to have for the customer service, we can start creating the contract by clicking on "Create New API".

Basically, what you need to do is pretty simple

Create the API(Service)
Create the data definitions (if any)
Add Paths, and define parameters, operations and return responses to the path

Enter the name of the service, and would be nice to add some of the description to it so it's easier for people to pick up what everything means.

Set the customer definition, show the detail of info we are going exchange.

Add and define the properties and their data type.

We can start adding the Paths to the doc,

Adding Parameters and define it's type.

And responses too, (if your path needs it)

Once we are done, it's time to export the API Standard documents.

Generating Fuse Project from Standard API Doc

Go to JBoss Developer Studio, create a new Fuse project by right click on navigation panel, select New and Fuse Integration project.

Provide a name to the project,

We are going to do this in a microservices approach, so select the most popular runtime - Spring Boot, and we will be running this on Cloud Platform - OpenShift.

Select the Spring DSL template.

You will have a sample Fuse project,

Add the generated API Spec doc to directory: src/spec/

Go to the pom.xml file, place the following in it,

<plugins>
....

<plugin>

  <groupId>org.apache.camel</groupId>

  <artifactId>camel-restdsl-swagger-plugin</artifactId>

  <version>2.21.0</version>

  <configuration>

    <specificationUri>src/spec/MyCustomer.json</specificationUri>

    <fileName>camel-rest.xml</fileName>

    <outputDirectory>src/main/resources/spring</outputDirectory>      

  </configuration>

</plugin>

....

</plugin>

Generate the XML by running following in command line tool,

mvn camel-restdsl-swagger:generate-xml

You will find a newly generated camel context named "camel-rest.xml", and it has all the Path implementation in Camel.

Copy everything in between "rests", into the original "camel-context.xml" and place in-between "camelContext". and add the following rest configuration on top of the rest component.

<restConfiguration apiContextPath="api-docs" bindingMode="auto"

            component="undertow" contextPath="/customer"

            enableCORS="true" port="8080">

   <apiProperty key="cors" value="true"/>

   <apiProperty key="api.title" value="Customer Service"/>

   <apiProperty key="api.version" value="1.0.0"/>

</restConfiguration>

Delete the generated "camel-rest.xml".

Mocking the APIs with Camel

We will mock the return result by adding a constant defined bean in the camel context.
Add a beans.xml under src/main.resource/spring (Right click on the folder, select New->beans.xml File)

Add the follow code snap to the beans.xml

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

 xmlns:util="http://www.springframework.org/schema/util"

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="        

    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd        

    http://camel.apache.org/schema/spring       http://camel.apache.org/schema/spring/camel-spring.xsd

    http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">

<util:list id="CustomerList" list-class="java.util.ArrayList">

   <ref bean="Customer"/>

</util:list>

<util:map id="Customer" map-class="java.util.HashMap">

   <entry key="name" value="Christina"/>

   <entry key="age" value="28"/>

   <entry key="contact" value="765483921"/>

</util:map>

</beans>

Add the camel route in the to "camel-context.xml", the first one returns a mock customer data, the second one takes the input of a customer info.

<route id="rest1-route">

  <from id="restone" uri="direct:rest1"/>

  <setBody id="route-setBody1">

    <simple>bean:CustomerList?method=get(0)</simple>

  </setBody>

</route>

<route id="rest2-route">

  <from id="resttwo" uri="direct:rest2"/>

  <log id="input-log" message=">>> ${body}"/>

    <setBody id="route-setBody2">

      <simple>Customer created</simple>

    </setBody>

</route>

And in the src/main/java/org/mycompany/Application.java added the new beans.xml file.

@ImportResource({"classpath:spring/camel-context.xml",

 "classpath:spring/beans.xml"})

public class Application {

Now, it's time to setup the dependency libraries into pom.xml.

<dependency>

  <groupId>org.springframework.boot</groupId>

  <artifactId>spring-boot-starter-undertow</artifactId>

</dependency>

<dependency>

  <groupId>org.apache.camel</groupId>

  <artifactId>camel-undertow-starter</artifactId>

</dependency> 

<dependency>

  <groupId>org.apache.camel</groupId>

  <artifactId>camel-jackson-starter</artifactId>

</dependency> 

<dependency>

  <groupId>org.apache.camel</groupId>

  <artifactId>camel-swagger-java-starter</artifactId>

</dependency>

and remove the tomcat dependency (we are using undertow now)

<dependency>

  <groupId>org.springframework.boot</groupId>

  <artifactId>spring-boot-starter-web</artifactId>

</dependency>

Finally, it's time to test. run the follow in command line:

mvn spring-boot:run

And two endpoint will be exposed for testing.

Christina Laptop$ curl http://YOURIP:8080/customer/id/123

{"name":"Christina","age":"28","contact":"765483921"}

Christina Laptop$ curl --header "Content-Type: application/json"   --request PUT   --data '{"name":"Christina","age":28,"contact":"765483921"}'   http://YOURIP:8080/customer/add

"Customer created"

You are now ready for the consumer to start testing the APIs.
Next part I will take you through how to actually implement the API, expose it on Cloud and manage it.

Red Hat Fuse 7 - Announcing Red Hat Fuse 7 GA!!

2018-06-04T14:30:00.000+08:00

Red Hat Fuse 7 (formerly called Red Hat JBoss Fuse) is now officially available. This cloud-native, distributed solution allows developers to easily develop, deploy and scale integration applications. Architects can compose and orchestrate microservices with Red Hat Fuse to introduce agility to the systems. In this release, Fuse also empowers integration experts and business users to become more productive with the self-service low-code platform. With this new agile integration solution, enterprises can now engage in wider collaboration with and among partners at a much quicker pace.
Here's where you can download it: https://developers.redhat.com/products/fuse/download/.

What's in Fuse 7?

Camel still is the core component in Fuse. Due to the complexity of integration solutions, a pattern-based integration solution is crucial for preventing developers from falling into complex, tangled, and spaghetti-like integration logic. Therefore, pre-built components and pre-defined patterns are always going to be the way to develop integration solutions, regardless of whether they are local or in the cloud.
Narayana is now the transaction technology, to get you ready for handling more-complex distributed transactions.
Jetty is deprecated and replaced by a more streamlined, lightweight web container: Undertow.
One of the most common questions I get from people just getting started with Fuse is, "Why do you need to have so many different options for runtime?" It's because we believe developers should be able to choose what best suits them, for example:

Spring Boot for microservices
Karaf 4 for OSGi lover
Red Hat JBoss Enterprise Application Platform for Java EE developers

A Cloud-Native Integration Platform

Building cloud-native applications can be complicated. Integrating them as well as making the integration solution itself be cloud-ready is even more challenging. You need to address the entire software lifecycle as whole, for example, creating applications with a smaller footprint, automating the deployment flow, and monitoring applications in a distributed environment. Because the Fuse 7 platform's features and technology that take care of much of this complexity, integration teams can now focus on the most important thing: the business logic of the integration application.
Lightweight Development
Building a small-footprint microservice/service is the most fundamental thing when it comes to moving integration into the cloud. Both the Spring Boot and Karaf runtimes are lightweight and microservices-enabled. Developers can take advantage of pattern-based integration in Camel, and plugins help developers package their integration application as a container to be cloud-ready.

Hybrid Deployment
After deployment, Fuse 7 helps developers to package the application into immutable images. Based on your versioning and deployment strategy, these images can be tagged and sent along to be deployed in different stages of the environment. The platform also takes care of the availability, scalability, and resiliency for the application; scaling up and down the application require just a few clicks in the UI. Or, if you prefer to automate that, you can. The application not only runs in the cloud, it can also be run as a single, standalone application. All three runtimes are available to run in both scenarios.

Management That Provides a Centralized Application View
Managing a distributed cloud environment is a nightmare: instead of a couple instances, you now have thousands of individual running instances. Fuse gives you a centralized viewing console in the cloud environment, so it will be much easier to locate all the running Fuse instances. Also, each running application is managed by Red Hat OpenShift, which provides comprehensive features for managing applications in the cloud.

Low-Code Integration for Citizen Integrators
Another new feature introduced in Fuse 7 is the Red Hat Fuse Online self-service platform, which allows citizen integrators to build integration services on their own. Fuse 7 allows a company to have a single, standardized platform instead of having multiple pieces of technology for different user characteristics, allowing maximum collaboration between developers and citizen integrators. With the intuitive interface and the highly customizable components' capabilities, citizen integrators can quickly deliver an integration solution without waiting for a developer's implementation. And developers can concentrate on developing more-complex integration logic and also package that into the platform so the citizen integrators can re-use it on the fly.
When you log in to the platform, you will be greeted with an overview of all the available integrations and connectors as well as the status of the running integrations.

Integrations allow an integrator to create an application by simply dragging and dropping the pre-built connectors or by mapping between input source and output target data.

When there is a need for more-complex process, you can also add customized connectors by simply importing the project package into the system. The customized connectors can be created by the in-house developer or provided online from our open source community.

Try Fuse 7 Now

You can try the new version of the platform here:
https://www.redhat.com/en/explore/fuse-online
So what are you waiting for? Start your new Fuse 7 adventure today.

Red Hat Fuse - Announcing Fuse 7 Tech preview 3 release.

2018-02-27T20:00:00.000+08:00

Red Hat Fuse 7.0 technical preview three is out today! On the pathway to become one of the best cloud-native integration platform, Fuse gives developer freedom to choose how they want to develop the integration solution, where they want to deploy it and capabilities to address new integration personas that do not have development experience.

By supporting the three major runtime, developer is free to work on the runtime of their choice.
By supporting standalone and cloud deployment, it simplifies the complexity to distinguish between these environments, allowing application to deploy freely among the environment of your choice.
All levels of developers are welcome, you can either dive deep into creating customize complex integration logic, or using the new low code platform to quickly build a simple integration.

In this Tech Preview release you get it all.

Fuse Standalone
- Spring-boot for microservice
- Karaf 4 for OSGi lover
- JBoss EAP for JavaEE developers
Fuse on OpenShift
- Plugins for easy containerize application
- Flexible cloud platform (based on OpenShift(Kubernetes) technology
Fuse Online
- Low code integration development
- Supports customization

Fuse Online Highlights

In this release you can now integrate with existing SaaS API. Simply by define a connector via importing Swagger documents into Fuse Online.

1. Upload Swagger document in the customize API Client Connector

2. Create a connection base on the customized connector

3. Get roll'n and start your integration!

Or customize your own integration by uploading an extension.

Watch and follow my video on getting started with this new feature.

Learn how to integrate an existing SaaS API with the Fuse LOW-CODE platform.
No coding is required.

Swagger Doc:
https://raw.githubusercontent.com/weimeilin79/fuse7tp3demo/master/waterpayment.yml

Water Company Dashboard:
https://water-company-tp3demo.4b63.pro-ap-southeast-2.openshiftapps.com/main

I hope you like what you see, go ahead and play with it. Let us know what you think!
CLICK HERE! To get started on Fuse Online!

The demo instructions can be found here:
https://github.com/weimeilin79/fuse7tp3demo/blob/master/README.md

Also adding a bonus demo:

The bonus demo instructions can be found here:
https://github.com/weimeilin79/fuse7tp3demo/blob/master/README-1.md

Red Hat Fuse - Tech Quickies series - Zero. Hello world

2018-01-03T23:37:00.001+08:00

This is the sample that I create for the hello world.

Prerequisite

Before you start, make sure you have installed the following software:

Setup your Development Environment

Download and launch the installer

For Mac / Windows user:

Download Red Hat JBoss Developer Suite here. Go to the folder where you downloaded the installer and double click on the downloaded file and kick off the installation.

For Linux user:

Download Red Hat JBoss Developer Studio 11.x Stand-alone Installer from JBoss Developer Studio Software Downloads. For detail installation instruction, click here.
Click Next on the Welcome page and choose your installation path and Next

Check components for complete Fuse environment:

Red Hat Developer Studio
Red Hat JBoss Fuse Tooling
Red Hat JBoss Fuse on JBoss EAP
Red Hat JBoss Fuse on Apache Karaf

Click Next, Enter credentials and click “Download & install”

Wait until components are downloaded and installed

Click “Open Red Hat JBoss Developer Studio”

Create a New Fuse Project on OpenShift

We are going to use the built-in SpringBoot on OpenShift template for our application. Right click in the Project Explorer pane and select New > Fuse Integration Project.

On the Choose a Project name page, enter a project name in the Project Name field. Click Next to continue and accept the defaults for all the other fields.

On the Select a Target Runtime page, select Next to accept all the defaults.

Select the latest version available for Camel.

The Advanced Project Setup page is where you select the pre-defined SpringBoot on OpenShift template.

Select Use a Predefined Template, Expand Fuse on OpenShift. Select SpringBoot on Openshift.

The Project Type will be automatically set to Spring DSL.

Click Finish.

The build process will be displayed at the bottom of the Advanced Project Setup page. When the build process is complete, you can see the new project in the Project Explorer pane.

Deploy and test your application

Organizing microservices - Modern Integration

2017-08-02T00:18:00.000+08:00

Microservices is probably one of the most popular buzz among my fellow developer friends, and I do like the concept of being flexible, agile and having simply having more choices. But as a person that worked in the software integration space for years, I started to see some resemblance of the old ESB days.

Looking at the problem from ten thousand feet up. A decade ago, we had to come up with a better way of organizing the spaghetti connection in between systems, stop duplicating effort on the same piece of business logic. That is when Service oriented architecture(SOA) became popular. By modularizing services, sharing them among others systems, and organize ways of communication, routing of data. And ESB is one implementation of that, maybe not necessary how it should be done.

I was very fortunate to participate in many of these kind of integration projects, and lead some myself. We worked with various middleware vendors, at the time the solutions was all about ESB. By removing the complex interconnecting logic between systems into ESB, negotiating data model among teams (with application/services leaders), setup WSDL contracts for webservices and adding BPEL processes if needed between calls. And I had seem a fair share of amount of these ESB myself, outside of ESB the everything seems very organized. But lift up the ESB “box” you will find many of them end up with even worst tangled spaghetti code (or diagrams depends on your tooling). And that giant bundle of chaos, then become the bottleneck in delivering changes in enterprise.

(On the side note. This is when I started on the lightweight ESB idea, and how I was introduced with Camel, Karaf, servicemix because it solves my problem of being to independently bundle my integration code and breaking that ESB box into smaller distribution..and so on..)

When the idea of independently deployable microservices came along. This giant monolithic monster with idea of smart pipe where it tries to do too much, like data model definition, process flow, interconnect protocols and data format transform ...etc and dumb endpoint where services just waits to be activated by events was soon cast aside by developers. The relation between each services are still messy and the monolithic nature make the cycle of development long and cumbersome. The microservice allows developer to be more agile, who needs that piece of “Integration”. So this where the resemblance of a problem where I once trying to solve :)

And let alone we still need to interact with other services(systems/applications) outside microservices. So learning from past here are the things we know what NOT to do..

No more large monolithic bundles application to speed up the dev cycle.
Stop doing complex business process inside system integration code
Data model shouldn’t be defined in the integration layer, it only leads to extra negotiation between teams.
State should not be keeped in the integration layer to allow maximum scalability
Create fix complex contracts between services.

And still there are things we liked about the old "integration" way.

Organized visualized interconnect view of systems.
Patterns that are already defined for integrating applications such as aggregations, separation, normalization of data.
Event-based approach. A more reactive way and less coupled ways to integration systems.

And that is why I came up with the layered architecture for microservice. The main goal that I wanted to achieve in the new integration architecture in the modern integration/application development is flexibility. Not only in the scalable way but also allowing developer to make any architectural change with ease.

So first thing first, in the new modern agile integration, the integration code itself should be sorted into different small services that make sense in the business world. And are decoupled and made them to be independently deployable microservices units. Then we can start apply true CI/CD to the integration code. This will avoid becoming a large monolithic ESB.

In order to bring some logical sense to my microservice spaghetti, and avoiding repeating the mistake of taking on too much in one single integration bundle. I have defined four layers for my microservice, so each will have it’s own responsibility making it easier to adopt changes.

Gateway layer Provides simple gateway routing capability such as versioning, dealing with different platform of devices.

Originally I want to divide this layer into two, but decide not to because the gateway pattern has become so popular now, people has the perception of gateway should achieve certain ability. (That is a whole other story I want to talk some other time). That is why I have two separate form of the gateway in my diagram, one represents as green other blue.

It’s all about the separation of concerns. This layer is all about generating putting together what is needed for external API consumption. This is where and when separate application domains tries to integrate with each other. And route to the correct versions of services. And clearly the two major concerns were,

Accessing policy rules, versioning of endpoints. (APIs)

Sets traffic throttle applying limits and policies
Security of APIs
Routing to correct versions

Combine, transform return result to client.

Citizen developer are the main creators here, they uses the capability provided by each microservice team to put together meaningful service to external clients.
Collect, split, or normalized a canonical data format to external users.
Translate data output depending on the clients
Routing between application domains

Composite layer Important middle tier that handles composition of multiple microservices. They do more complex routing from processing the content data and aggregates/split data and populate the split/aggregated result to other microservices by triggering events or simply passing it. This layer hides the complexity of microservices away from clients.

This layer is probably the layer that does most of the old time integration logic where it is responsible for

Composing microservices

By calling the API available from microservices, and transforming data as needed between them and routing data to the corresponding microservices based on it’s content.

Triggers events within domain

I believe strongly that event base does best at decoupling stickiness between each service. And allowing performance at it’s best because of its asynchronous nature. The event bus here doesn’t necessary has to be a message broker, but any forms of Bus.

Enrich contents

The content enrichment here should be specific to the format and avoid any business side enrichment when possible.

Applying Integration patterns

Old integration problem will not just go away in microservices world. By applying proven patterns, we can avoid making mistakes.

Caching

The REST architecture specifically talks about a caching layer. It make sense for the composite layer to try apply caching mechanism because it’s non business related, but do make significant impact to the performance and fault tolerance ability of the system.

The composite services should also be included in the application domain, which I will talk about in the basic layer.

Base layer Like the name which is most likely represents the basic components of the system. Handles data retrieval or business logic processing. Each should be independent and self-contained.

Borrowing from the idea of Bounded Context, in this case logically organized group of microservices into what is called application domain. Where each domain represents separate entities of business functions that shares the same data model, this eliminates the need for doing too much transformation in the code.

Anti-corruption layer - This layer handles interface to legacy application or anything that work against microservice quick and flexible principle. This layer is built in protection wall to your system, by doing the transforming job and translate between two very distinct implementations of the system.

Till next time! :)

Reference architecture of Agile Integration

2017-07-26T00:04:00.000+08:00

OK. I said it. Integration is still around, but in different form from my last blog post. So what does modern integration looks like? Looking at how agile scrum has taken over traditional waterfall development framework, by enable shorter deliver cycles, faster feedback, and having the flexibility to rapidly adapt changes. I believe it’s time for traditional integration to be agile again. By breaking up traditional ESB into distributed microservices.

A little recap of what should be in Agile integration:

Distributed Integration

Lightweight, support distributed deployment
Pattern Based Integration
Reusable Connectors
Microservices

Containers

Cloud native solutions
Lean artifacts, individually deployable
Container based scaling and high availability

Well defined, re-usable, and well managed end-points
Ecosystem leverage

Base on these three principle, I was asked to create a reference architecture for it. After giving some thoughts into it, drawing up numerous boxes and diagram, something clicked when I watched a 3 year old playing. And here, this is my answer…

Yup. That is it! Reference architecture of Agile Integration. Architecture is often needed during the design phase of a software project, it provides a structure and backbone for the software project. Once it’s done, it’s done, any architectural modification will be difficult and can bring disastrous result. We all knew “one thing will never change in software development is Change”. Maybe it’s because regulation requirement, market demands or simply learning more about the business domain. So I started to think, can we build an architecture that is flexible for change, and can be shaped into the needs of a project as needed. So base on that concept, this is my reference architecture for Agile Integration, the modern integration application development that enables flexibility from many angles.

There are the major components in my reference architecture diagram for Agile Integration:

PaaS - Provides a foundation for fast pace software development by allowing developer to self-service, as well as enable automated provision.It will also boost operational efficiencies by making the entire environment DevOps ready.

Security/Identity management(IAM) - Handles basic authentication and authorization for application interfaces and platforms.

Automation - Automates both the processes of building applications, and rolling deployments strategies for upgrades. Together with continuous integration tools, it can achieve continuous delivery of the application softwares.

Logging and Tracing - What was suppose to be easy in monolithic world, has now become one of the challenge in distributed microservice architecture for integration. As it has now become harder to have a unified view of how things are doing. It is important to have a way to see logging as a whole. And has tracing capability for different activities.

Containerized packages - Container technology allows us to build immutable portable lightweight application package, that is language agnostic. The configuration are independent from this builds allowing same package to deploy quickly in multiple environment.

Container management - In large scale system it is essential to have a convenient management platform to manage all the containers running the applications. Taking care of the monitoring, discovery, recovery and failover of the running containers.

Microservices - Large applications/services are broken down into easy maintainable pieces and can be independently develop and deploy in distributed environment. Each of them should be built with failure in mind. Microservices

Load Balance/Service discovery/Network management - Microservice are built for flexibility, according to current system load, the load should be automatically balance to the each running instance. All services should be registered and can be found in the system without complex setup. Ideally a proxy endpoint for external users to hide away the deployment complexity.

Layered architecture - Logically organize microservice into a sets of layers, each has it’s own responsibility to avoid duplication of effort and make it easier to replace later on. Currently there are 4 layers

Gateway layer - provides simple gateway routing capability such as versioning, dealing with different platform of devices.
Composite layer - important middle tier that handles composition of multiple microservices. The do more complex routing from processing the content data itself and sometimes handle more complex data aggregation or normalization.
Base layer - like the name which is most likely represents the basic components of the system. Handles data retrieval or business logic processing.
Anti-corruption layer - This layer handles interface to legacy application or anything that work against microservice quick and flexible principle. This layer is built in protection wall to your system, by doing the transforming job and translate between two very distinct implementation of the system.

Application domains - This one is more like a pattern then architecture. But since we mention about layers in software I also wanted to touch a little bit on this. Because it’s all about reining the microservice into a more organized way. It’s a good practice to have each application domain defined (using same set of data model). And have other external integration microservice to wire between each domains. One other good thing about microservice is that you have the flexibility to move each small microservice to different domains as which it fits better with.

API management - Managing the APIs by reinforced access policy, collecting statistics around the usage of APIs. Also building a healthy eco-system among the developer and users of APIs. Further to allow direct revenue source from the APIs.

There are certainly some limitation in today’s technology to allow a complete pliable architectural form. But I believe this model can bring flexibility into the system without significant impact to the system when changes occur. I will explain more details in my upcoming posts.

The rise of Agile Integration, Integration is not DEAD nor LAME!

2017-06-06T12:26:00.000+08:00

I wanted to blog about this for a very long time, but because of work and being too lazy to turn on my laptop on the weekends, now I finally have a couple of hours to sit down and start blogging.

"Integration is DEAD."
"No, no, let's talk about microservice!!"
"What is that again? Integration, you mean SOA? It's lame... ".

These makes the majority of the percentage when I talk about Integration. In fact, if you happen to be in any the software conference, you see the rooms full when it's about "container" and "microservice." I understand the nature of a developer, always seeking for the latest and greatest technology out there. BUT!! In my opinion, I don't think Integration is going AWAY if you containerize your application. Neither does it will DISAPPEAR in a microservice architecture. In fact, Integration just gets more complicated and tricky when you try to break an application into smaller pieces and have them running all over the place in the distributed environment!

It was hard to deal with when we have to orchestrate, choreograph and compose all the coarse-grained services together, and imaging letting tinier, smaller and more refined services talks to each other. Yeah, trying to get an oversized heavy-weight strong bull to co-ordinate and communicate is one thing. I don't think to have to conduct thousands of small cute duckling, (I know, they are now smaller to handle one by one, light and easy to replace) is going be any easier. :-)

Maybe the word Integration was abusively used by too many vendors (Us too ;-D ), and it is too often associated with ESB, which IMHO should be one of the Integration strategies. ESB can be overly complicated with a wrapped around RPCs. Integration is about connecting, composing between services, neglecting integration when implementing microservice architecture (MSA) is going to be disastrous. I am sure a lot of smart people out there designing the MSA has already taken this into account. But they probably did not realize that because Integration is already deeply ingrained into their minds, even in microservice we still want loose coupling. Again, in my opinion, the wall between application development and integration are getting thinner and thinner, by that I mean you do integration in microservice way and apply integration in your microservice. So, yes, integration needs to change, adapt and adjust accordingly but it never goes away:

Current deployment method

As now services are updated more frequent than ever before, we need to quickly respond to the change, in the integration layer. We can package it into the container and standardize the deployment method across the company.

Speed and automated delivery

A quicker respond to developer, continuous integration and delivery, faster time to markets can be achieved by adopting the DevOps trend. Integration should also be part of this cycle. Not OUTSIDE!

Flexible for changes

Making sure the message can be route to the correct service, allowing automatic discovery of services, with load balance capability that shares the load no matter it's synchronized or event base call. Providing and compose with flexible contracts like APIs between service can also minimize the impact on the changes.

More robust towards failure

In the integration, it's all about the isolation of failure, and handles the errors, increase service resiliency. Also the ability for it to adopt version changes, and service lifecycle management is also one of the important factors in Integration.

Some concepts of doing integration in MSA comparing with monolithic application are poles apart. For example, continue from the bull and duckling theme. When you try to get a couple stubborn, massive bulls together and adjust position takes a lot of energy, and often you will need to be creative and smart to improve according to their position. Therefore the pipelines have to be smart and do many manipulations. Whereas the ducklings are easy to move around and cheap to replace. If one got sick, naughty or does not have the right color you need, simply toss it and get a newly hatched egg. (Alright, I don't mean any animal cruelty here, it's just a metaphor. ). The pipe that brings these microservice can be dumb or simple. Because each duckling is cheap to create.

In my eye, microservice should never go freely create on wherever and whenever it should be at least logically bound. I call this application domain. By circling these microservice, we will be able to see a much clear picture of inbound microservice integration and outbound microservice integration.

The inbound microservices integration will be a lot more flexible in protocols, data formats because of the possibility to connect to various datasources and SaaS application. Since they are bounded within the same application domain, it's likely the data models are shared between these microservices. They tend to be more coupled with each other (although service autonomy is where we are trying to get. They are probably handled by the same person.) There will be a couple of microservices responsible for composing some of the other microservices together too. The logic inside should be simple and straight forward. And each of the microservice should easily take care of the protocol and data extract itself.

On the other hand, the outbound microservices integration should be strictly loosely coupled with inbound microservice. Preferably access commonly managed endpoints from each application domain, yes, I mean APIs. It is mostly device or client bound. As it often required to put together a more coarse-grain service like the old ESB days for the client application or from a process.

Both inbound and outbound integration can be synchronous and asynchronous, all depends on the need of your system. So don't limit yourself to just one option. I strongly recommend the Enterprise Integration Pattern Book, as I said, the integration problem is still there. The patterns are extremely useful in this manner.

Agile Integration is Red Hat's opinionated way of doing the integration. And most importantly it's backed up by Fuse (Apache Camel) and OpenShift.

It talks about 3 main pillars, which resonate my integration believes in MSA world.

Distributed Integration

Lightweight, support distributed deployment
Pattern Based Integration
Reusable Connectors

Containers

Cloud native solutions
Lean artifacts, individually deployable
Container based scaling and high availability

Well defined, re-usable, and well managed end-points
Ecosystem leverage

So! I hope my murmur on this blog post will at least makes you think "Integration is NOT LAME". And start thinking about it when next time you create your next microservice.

Red Hat JBoss Fuse - Applying API Best Practices in Fuse

2017-01-12T01:07:00.000+08:00

API plays a huge part in modern integration architecture design, a good design will allow your application to thrive, a bad design will end up on the cold stone bench and eventually vanishes :(....
Well, to avoid this tragic happens to our APIs, there are certain guidelines that we might want to consider to follow, I know there are lots of debates out there on the best practice of API design, and I don't think it will ever end. It is really depending on many different factors, mostly dominate by the size and complexity of the integration solution, and the company culture. And many of them are related to how to manage it instead of designing. (Of course there are many others like API security, how to do versioning and all these sorts of things. These are more closely related to API management, that I will not cover in this post. )

This is what I think is a good API,

1. Intuitive- It must be easy to understand and use without documentations.
2. Stable- Not only it should be running but with good performance too.
3. Demands - Creating useful functionally, no matter how nicely your API is documented, how easy it is to use, it people don't need it, they won't call it.

The second one, is mostly depend on how robust your code is, the DEVOPS process, how the IT infrastructure is built and so on. And third one is a much harder question to answer, so I am going to skip these two first and focus on ONE, Intuitive. Here are just couple of best practices of that I think that can make APIs more intuitive , and how it's done in Camel. (RESTFul API for strictly speaking, the technology the most people use is the standard.)

Simply but concrete naming for the URI.

Common rules, use nouns instead of verbs, that describe the content the API is providing, such as customer, account and product etc. In Camel the place to define it is in the REST DSL, it allows developer to define REST services in it's application. Define the name of the API in the URI and have several layers of API description.

<get uri="customer/{customerid}">

<to uri="direct:getCustomerinfo"/>

</get>

<get uri="product/{id}">

<to uri="direct:productInventory"/>

</get>

<get uri="account/profile/{acctid}">

<to uri="direct:getprofile"/>

</get>

From the architectural stand point, one of the good thing about Camel and Fuse/Fuse Integration Service. You naturally create these independent chunks of service, that either connects to a datasource (or multiple if you want, but that does not quite fit in the sprits of doing microservices.) and then becomes a microservice. Or you can use it to create a service that compose/orchestrate APIs for the consumer.

Use HTTP Method for CRUD if possible

Instead of creating four different names with the verbs that we normally do in webservice back in the days, take advantages of the HTTP method, that is ready there and the HTTP header actually maps to different operations that you need to do, In Camel DSL, it is very simple to define it,

READ -> GET
CREATE -> PUT
UPDATE -> POST
DELETE -> DELETE

<rest path="/account">

<get uri="profile/{acctid}">

<to uri="direct:getprofile"/>

</get>

<put uri="profile/{acctid}">

<to uri="direct:createprofile"/>

</put>

<post uri="transfer/{acctid}/">

<to uri="direct:dotransfer"/>

</post>

<delete uri="profile/{acctid}">

<to uri="direct:deleteprofile"/>

</delete>

</rest>

Not only it eliminates the whole team to agree upon naming of the actions, and since it's globally recognized standard, consumers will find it easier to understand.

Make the most out of HTTP Errors,

Wrapping your own error is good, but think about this, when a consumer gets response from the API, what is the first content it get? The HTTP response header, instead of processing the entire payload (not restricting to error, there are several successful status code too).
Here are the list of http status codes, there are a couple of then that are commonly used in RESTApi, For instance, here are some of the common one used in RESTFul API

HTTP Code	HTTP Methods	Description (From Wiki page)
200	GET, DELETE	Standard response for successful HTTP requests. The actual response will depend on the request method used.
201 Created	PUT, POST	The request has been fulfilled, resulting in the creation of a new resource.
202 Accepted	POST, PUT, DELETE	The request has been accepted for processing, but the processing has not been completed.
400 Bad Request	GET, POST, PUT, DELETE	The server cannot or will not process the request due to an apparent client error
401 Unauthorized	GET, POST, PUT, DELETE	Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided.
403 Forbidden	GET, POST, PUT, DELETE	The request was a valid request, but the server is refusing to respond to it.
404 Not Found	GET, POST, PUT, DELETE	The requested resource could not be found.
408 Request Timeout	GET, POST, PUT, DELETE	The server timed out waiting for the request.
409 Conflict	PUT, PUT, DELETE	Indicates that the request could not be processed because of conflict in the request.
500 Server Error	GET, POST, PUT, DELETE	A generic error message, given when an unexpected condition was encountered and no more specific message is suitable.
503 Service Unavailable	GET, POST, PUT, DELETE	The server is currently unavailable (because it is overloaded or down for maintenance). Generally, this is a temporary state.

In Camel, you can set the value in header to return the HTTP status code

<setHeader headerName="camelhttpresponsecode">

Setting the right granularity of data and use common data format,

I don't believe in ONE SIZE FITS ALL data model when it comes to providing information to various consumer, the amount of information can be shown in a mobile device screen, an smart watch or a desktop computer is definitely different.
Camel has capability for you to automatically transform the data from POJO into the format that you wanted by specifying the output, or you can simple so marshaling with various data format Camel supported or do a little extra processing with the Data Mapper transform component.

The binding mode:

Binding Mode	Description
off	Binding is turned off. This is the default option.
auto	Binding is enabled and Camel is relaxed and support json, xml or both if the needed data formats are included in the classpath. Notice that if for example `camel-jaxb` is not on the classpath, then XML binding is not enabled.
json	Binding to/from json is enabled, and requires a json capabile data format on the classpath. By default Camel will use `json-jackson` as the data format. See the INFO box below for more details.
xml	Binding to/from xml is enabled, and requires `camel-jaxb` on the classpath. See the INFO box below for more details.
json_xml	Biding to/from json and xml is enabled and requires both data formats to be on the classpath. See the INFO box below for more details.

Or DataFormat Marshaling (Don't forget to add the dependency of the converting libraries)
<marshal ref="transform-json">

</marshal>

Or Data Mapper transform (Dozer)
See my previous posts

Automatic generated documentation.

One of the nice handy little tricks when doing API in Camel with REST DSL, you can simply integrate it with Camel swagger java components. It will expose the API in swagger format.
In Camel, don't forget to first to add the dependency in your maven pom,

      <dependency>

          <groupId>org.apache.camel</groupId>

          <artifactId>camel-swagger-java</artifactId>

          <version>x.x.x</version>

          <!-- use the same version as your Camel core version -->

      </dependency>

     And inside the Camel context, when setting up the rest configuration, simply add the API related configuration into it.

<restConfiguration apiContextPath="api-docs" bindingMode="json"

component="servlet" contextPath="/demos">

<apiProperty key="cors" value="true"/>

<apiProperty key="api.title" value="Healthcare demo clinical API"/>

<apiProperty key="api.version" value="1.0.0"/>
</restConfiguration>

In summary this is the diagram that maps to Camel and best practices,

Thanks!

Red Hat JBoss Fuse - Getting Started with Fuse Integration Service 2.0 Tech preview

2017-01-05T06:15:00.003+08:00

I just realized that I did not do a getting started for Fuse Integration Service 2.0 Tech preview before I did the pipeline demo, thanks for those of you who reminded me! :)

To get started with FIS 2.0, for people who has just getting to know the technology, here is how I interpret it. Basically, it's divide into two aspect,

1. Integration development, FIS uses Apache Camel as the core technology that creates, orchestrate, compose microservices into a super lightweight thin integration layer, and become the API provider and service orchestrator through exposing RESTful or messaging service endpoints. And you can choose to either package and run it with Spring-Boot or Karaf.

2. Application Deployment and Management, FIS takes advantages of OpenShift platform, and allows you to separately deploy the micro-integration service among distributed environment, at the same time takes care of the failover, high availability, load balancing and service lookup problem for you.

So, now we know what is in FIS 2.0, it's time to take a closer look at how it is done, as a developer, you first have to decide to go with Karaf (OSGi) or Spring-Boot framework, I personally prefer the Spring-Boot, because it's matches more to the microservice concept of a lighter deployment package. But it is up to you, the developer, after all you are the god and creator of your application. After the framework is chosen, developer will start developing the micro-integration services, composing between microservices or even use it to create a microservice. (With two frameworks, the development experience of the route itself is basically the same, by configuring camel components. )

Once the developer is ready to deploy the integration service, we can then decide how to deploy it onto the OpenShift platform. In FIS 2.0 there are two options, Binary S2i will build the entire application locally, and push it onto OpenShift, so OpenShift platform will use it to create and build the container image that it will run on, and for Source S2i, everything is build on top of OpenShift, so developer need to set the location of the source code in order for OpenShift to retrieve it to build the application and the container image.

And that is all. It is actually much much more powerful, it's hard to describe it in one go, dive into it, and you will soon find how fascinating the technology is, and how it can help you to resolve your current problems.

Here is a quick video that shows you how to get your first FIS 2.0 running.
The steps are as follows,

Install and start up OpenShift on your local machine
Install FIS image stream definition into OpenShift (raw.githubusercontent.com/jboss-fuse/application-templates/application-templates-2.0.redhat-000026/fis-image-streams.json)
In JBoss developer studio, create a Camel Spring-boot using the new archetype in FIS 2.0
(Archetype catalog url: https://maven.repository.redhat.com/earlyaccess/all/io/fabric8/archetypes/archetypes-catalog/2.2.180.redhat-000004/archetypes-catalog-2.2.180.redhat-000004-archetype-catalog.xml)
Deploy to OpenShift using the Binary S2i tool (maven plugin)

Red Hat JBoss Fuse - Automate integration CI/CD processes

2016-12-23T02:58:00.001+08:00

Long time no see :) , for many of you who knows me, I've been through a major change in life, moving to another country for couple of years, that is why there are less updates in my blog. But, I am back! Getting back to business. Fuse Integration Service 2.0 Tech preview was out two weeks ago, since this version is base on OpenShift 3.3, which has pipeline capability prebuilt on top of it (Tech preview on OpenShift as well). This get it one step closer to a more automated and agile continuous integration, and deployment one-stop platform for us, the integration developer.

For the pipeline to work on OpenShift, you will need Jenkins installed and running. OpenShift uses it to build, process and handle all the workflows. If you are familiar with developing in openshift, to build the pipeline is pretty simple and straight-forward. The pipeline is defined as a build configuration in OpenShift, just create a build config then import it to the namespace you want it to be in. And that is it.

This is what the build config looks like, note the strategy type is called JenkinsPipeline, this will trigger the interaction with Jenkins, and pushes the defined Jenkinsfile onto the server itself. The Jenkins Server will then interact with openshift and start the automated CI/CD process.

kind: BuildConfig
apiVersion: v1
metadata:
name: pipelinename
labels:
name: pipelinename
spec:
triggers:
- type: GitHub
github:
secret: secret101
- type: Generic
generic:
secret: secret101
strategy:
type: JenkinsPipeline
jenkinsPipelineStrategy:
jenkinsfile: "
node('maven') {
stage('build') {
print 'build'
openshiftBuild(buildConfig: 'buildconfigname', showBuildLogs: 'true')
}
stage('staging') {
print 'stage'
openshiftDeploy(deploymentConfig: 'deploymentconfigame')
}
}"

As you can see on the above Jenkinsfile in the build configuration, it's interacting with the openshift itself through the OpenShift and Jenkins plugin. For instance, you could trigger to build an image, deploy the application through calling the deployment config, tag an image or even scale up and down the number of containers.

This upper part of the blog is pretty generic to most of the application running on OpenShift, but, HEY, Fuse Integration Service is just another application running on top of it. But this application just simply contains PATTERN BASE integration technology that has 160+ built-in components in it, so we smart developer don't have to waste time and energy on repetitive stuff, no big deal. :)
No matter what version you are using, this pipeline capability can help you automate your integration microservice.

Here is a small demo video that takes you through the entire process.

Red Hat JBoss Fuse/A-MQ - Fuse and A-MQ Version 6.3 GA is released!

2016-10-06T23:00:00.000+08:00

Fuse and A-MQ 6.3 GA has just went out. Maybe, you would think this is just only a minor version release why should I care? Hold your thoughts on that! Because they have done a lot of improvements and also added many new features into this release.

Besides various bug fixes and making sure Fuse Fabric is much more stable. There are two major change in this version update:

New Tooling in JBoss Developer Studio (JBDS) 9.1 GA.
Newer Apache Camel version – Camel v2.17.

I was really impressed by the work put in to make developing Camel application much simpler. First is the installation of tooling itself. Now it has a all-in-one installer so you don't need to worry about which plugins you need to check. See the videos below to see the new "Getting Started" of Fuse 6.3.

And If you notice from the above video, the presentation of camel route in JBDS has also updated. It fixed some of the miss representation of logic and making it easier to read.

Old Camel Route

New Camel Route

On the right hand side, there are more components added to the component palette.

In this new GA version, it has supported couple of new connectors,

ServiceNow

ServiceNow offers everything-as-a-service cloud computing, Camel provides access to all of ServiceNow REST API. You can manage and automate your IT service with Fuse.

Kafka

Apache Kafka is publish-subscribe distributed messaging broker, a very popular messaging alternative, with Fuse's new component can now listen and put data into Kafka.

Spark

Apache Spark is an cluster computing framework, provides programmers with a centered application programming interface.And a read-only multiset of data items distributed over a cluster of machines, and maintained in a fault-tolerant way. To integrate with Spark, Fuse provide a bridge between Camel connectors and Spark tasks, and dynamically choose a task to execute.

Paho

Eclipse Paho project provides open-source client implementations of MQTT messaging protocols, this connector integrate to this client using the Eclipse Paho library.

Slack

Slack is a cloud-based team collaboration tool, it's has IRC-like features, persistent chat rooms (channels) organized by topic, private groups and direct messaging. The connector allows you to connect to an instance of Slack and delivers a message contained in the message body via a pre established Slack incoming webhook.

Kubernetes

Kubernetes is implemented as container cluster manager, provides mechanisms for deploying, maintaining, and scaling applications on Docker. This connector allow you to interact with Kubernetes operation, and use it to manage and integrate with your containers.

PDF Documents

This components provides the ability to create, modify or extract content from PDF documents. This component uses Apache PDFBox as underlying library to work with PDF documents.

Also with mentioning is the addition of swagger-java module, with the REST DSL in Camel, we will now be able to build a complete, flexible API-Based application with Camel being the API connector base technology. Using swagger-java module to automatic generate the documentation.

Take a look and play with the new Fuse 6.3!

Red Hat JBoss A-MQ - A-MQ 7 alpha is ready!

2016-07-15T21:41:00.001+08:00

I am sure this is the moment we have all been waiting for a long time, JBoss A-MQ 7 is finally out and is currently in Alpha. A-MQ is a fast and flexible messaging platform that reliably delivers information and easily integrates the various components in your application environments.

A-MQ 7 consists of several new components and features, it is base on Project Apache ActiveMQ Artemis. The code base was donated by HornetQ. Having high performance journal base on NIO implementation, allows A-MQ 7 to become a fast , high performance and reliable messaging platform. Rather than relying other persistence storage it is very highly optimized for general message use cases. The message broker as always allows you to balance load across a cluster of brokers, the balancing of the message not only distributed the incoming message to brokers sequentially but also takes into the account of the number of consumers and selectors in the broker so the performance can achieved even distribution of the load.

The support for Cross-language and Multi-protocol in A-MQ 7 allows many more native connectivity from applications written in Java (JMS), C++, Python, JavaScript/Node.js, .NET. Multiple transport protocols for exchanging data with AMQP 1.0, MQTT, STOMP, OpenWire, HornetQ Core.

One of the bigger innovation of A-MQ 7 is the introduction of Interconnects, Interconnect is a message router you can use to build large-scale messaging networks. A-MQ Interconnect uses the AMQP protocol to create a redundant application-level messaging network. It works as an interconnect layer between messaging clients and their brokers. And why do we need it? The beauty of the interconnect allows you to solve the good old problems when we only have multiple brokers, there could be too many clients trying to connect to to a single point of brokers, one broker could have too many queues create, one of a broker handling too much throughput comparing with the others, have interconnect is to add another routing layer before your broker, it is flexible nature allows messages to flow through it and evenly spread the load. It automatically find the fasting routing path so the message will arrive efficiently. And since it's base on AMQP protocols, you can set your own selector on it base on your messaging routing logic.

And as always, we still have Hawtio as the management tooling, where you can see the details of how the brokers are doing now.

For those of you who uses the old version of A-MQ or HornetQ, A-MQ 7 support both A-MQ 6.x and HornetQ clients for backward compatibility.

To get your hands on the latest Alpha version of JBoss A-MQ 7, please go to http://developers.redhat.com/products/amq/download/