Chandana Napagoda's Blog

Kafka - A Quick and Simple Guide

2025-04-08T22:52:00.003+09:30

Introduction

Apache Kafka is a distributed event streaming platform. Originally developed at LinkedIn, it is now an open-source project under the Apache Software Foundation. Kafka is widely used for building real-time data pipelines and streaming applications.

What is Apache Kafka?

Kafka is a messaging system that lets applications publish (write), subscribe (read), store, and process streams of data in real-time.

Kafka Core Concepts

1. Producers and Consumers

Producers → Send data to Kafka topics.
Consumers → Read data from Kafka topics.

2. Topics and Partitions

Topic → Logical channel for a specific type of data.
Partition → Topics are split into partitions for scalability. Each partition stores data in order.

3. Brokers and Cluster

Broker → Kafka server that stores data.
Cluster → Group of brokers working together.

4. Zookeeper

Kafka uses Zookeeper to manage broker metadata and coordination tasks.

Note: Newer Kafka versions can run without Zookeeper (KRaft mode).

5. Replication Factor
This provides fault tolerance and state how many copies of each partition exist across brokers. as example if replication factor is 3, that mean partition will kept inside leader and 2 brokers

6. Partition Rebalancing

When consumer join or leave a consumer group, Kafka will triggers a rebalance to redistribute partitions among the available consumers. This will ensure load balancing, but it may temporarily stop message processing

How Kafka Stores Data

Kafka stores data in logs → sequence of records per partition. Logs are divided into segments (files on disk). This helps manage data efficiently and allows old data to be deleted to free up space.

Each record has a unique offset. Kafka assigns each record an offset as it is produced, which consumers use to read records in order.

Data can be replicated to multiple brokers for fault tolerance. Each partition has one leader broker and several replica brokers. The leader handles all reads and writes, while followers replicate the data.

Ensuring No Duplicate Messages

Idempotent Producers : Enable it by setting enable.idempotence=true we can ensures the same message is not written more than once even if retries happen.

Exactly Once Semantics (EOS) : Use transactional.id on producers and manage offsets manually in consumers(commit offsets only after processing the record).

Consumer Offset Management : By disabling auto-commit (enable.auto.commit=false) and committing offsets explicitly after successful processing. This will ensures at least once delivery. Combine with Idempotent processing logic for effectively once delivery.

How Kafka Handles Multiple Consumers the Same Group ID

Kafka uses consumer groups to allow multiple consumers to divide the work. Each consumer in the group reads from a unique subset of partitions. They share partitions of a topic. Kafka ensures only one consumer from a group reads a partition at a time.

Consumers in different consumer groups read the same data independently

How long does Kafka store data?

By Default, it's time-based retention (retention.ms=604800000), but this can change to size-based retention(retention.bytes) or forever (log.retention.ms = -1)

XSLT Date and Time Manipulation

2024-07-03T11:41:00.000+09:30

In XSLT (Extensible Stylesheet Language Transformations) 2.0 or later, we can perform several date and time manipulation processes. I thought of writing a blog post so others can easily find examples as well.

Basic Date & Time XSLT Functions:

current-date() - Returns the current date of the server that XSLT runs

current-time() - Returns the current time of the server that XSLT runs

current-dateTime() - Returns the current date and time of the server that XSLT runs

implicit-timezone() - Returns the implicit time zone of the server that XSLT runs

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0">
    <xsl:template match="/">
        <root>
            <currentDate>
                <xsl:value-of select="current-date()"/>
            </currentDate>
            <currentTime>
                <xsl:value-of select="current-time()"/>
            </currentTime>
            <currentdateTime>
                <xsl:value-of select="current-dateTime()"/>
            </currentdateTime>
            <implicitTimezone>
                <xsl:value-of select="implicit-timezone()"/>
            </implicitTimezone>
        </root>
    </xsl:template>
</xsl:stylesheet>

Date & Time Extracting XSLT Function:

year-from-date() - Extracts the year from a date.

month-from-date() - Extracts the month from a date.

day-from-date() - Extracts the day from a date.

hours-from-time() - Extracts the hours from a time.

minutes-from-time() - Extracts the minutes from a time.

seconds-from-time() - Extracts the seconds from a time.

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0">
    <xsl:template match="/">
        <root>
            <year>
                <xsl:value-of select="year-from-date(current-date())"/>
            </year>
            <month>
                <xsl:value-of select="month-from-date(current-date())"/>
            </month>
            <day>
                <xsl:value-of select="day-from-date(current-date())"/>
            </day>
            <hours>
                <xsl:value-of select="hours-from-time(current-time())"/>
            </hours>
            <minutes>
                <xsl:value-of select="minutes-from-time(current-time())"/>
            </minutes>
            <seconds>
                <xsl:value-of select="seconds-from-time(current-time())"/>
            </seconds>
        </root>
    </xsl:template>
</xsl:stylesheet>

Date and Time Manipulation Functions:

Please note Saxon doesn't supports add-yearMonthDuration-to-date and other related 3 operations. So here is my workaround to manipulate dates. Please refer my old blog post on this

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0">
    <xsl:template match="/">
        <root>
            <add-yearMonthDuration-to-date>
                <xsl:value-of select="current-dateTime() + xs:yearMonthDuration('P1Y6M')"/><!-- Add 1 year and 6 months -->
            </add-yearMonthDuration-to-date>
            <add-dayTimeDuration-to-date>
                <xsl:value-of select="current-dateTime() + xs:dayTimeDuration('P10DT2H')"/><!-- Add 10 days and 2 hours -->
            </add-dayTimeDuration-to-date>
            <subtract-yearMonthDuration-from-date>
                <xsl:value-of select="current-dateTime() + xs:yearMonthDuration('-P1Y6M')"/><!-- Subtract 1 year and 6 months -->
            </subtract-yearMonthDuration-from-date>
            <subtract-dayTimeDuration-from-date>
                <xsl:value-of select="current-dateTime() + xs:dayTimeDuration('-P10DT2H')"/><!-- Subtract 10 days and 2 hours -->
            </subtract-dayTimeDuration-from-date>
        </root>
    </xsl:template>
</xsl:stylesheet>

String Format Conversion Functions:

format-date() - Formats a date according to a given pattern.

format-time() - Formats a time according to a given pattern.

format-dateTime() - Formats a date time according to a given pattern.

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0">
    <xsl:template match="/">
        <root>
            <formattedDate>
                <xsl:value-of select="format-date(current-date(), '[Y0001]-[M01]-[D01]')"/>
            </formattedDate>
            <formattedTime>
                <xsl:value-of select="format-time(current-time(), '[H01]:[m01]:[s01]')"/>
            </formattedTime>
            <formattedDateTime>
                <xsl:value-of select="format-dateTime(current-dateTime(), '[Y0001]-[M01]-[D01]T[H01]:[m01]:[s01]')"/>
            </formattedDateTime>
        </root>
    </xsl:template>
</xsl:stylesheet>

Compare Dates :

When comparing dates with XSLT 2.0 or later, we can use default XSLT comparison operations such as 'lt' (less than), 'gt' (greater than), etc.

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0">
    <xsl:template match="/">
        <root>
            <dateCompareLessThan>
                <xsl:value-of select="current-date() lt (current-date() + xs:yearMonthDuration('P1M'))"/>
            </dateCompareLessThan>
            <dateCompareNotLessThan>
                <xsl:value-of select="(current-date() + xs:dayTimeDuration('P1D')) lt current-date() "/>
            </dateCompareNotLessThan>
            <dateCompareGreaterThan>
                <xsl:value-of select="current-date() gt (current-date() - xs:dayTimeDuration('P1D'))"/>
            </dateCompareGreaterThan>
        </root>
    </xsl:template>
</xsl:stylesheet>

You can use an online XSLT playground such as xslttest to verify the examples listed above.

Please refrain from copying content to other websites

Global Exception Handling With Spring Boot

2024-06-28T18:00:00.002+09:30

In a Spring Boot controller, things can sometimes go wrong. For example, an API consumer might not send the expected path parameter or header value. They might call the API with an invalid payload or encounter an unexpected runtime error. When these kinds of errors occur, we can use Spring Boot's Global Exception Handling mechanism to manage them and provide meaningful responses back to the API consumer.

@ExceptionHandler is used to handle exceptions in specific controllers classes and/or methods level. In a Spring Boot Restful service, errors can be handled in three different ways:

Using ControllerAdvice - This annotation allows global(central) exception handling across multiple controllers.
Using @ExceptionHandler at each controller level - For more granular control
Returning ResponseEntity with the appropriate status code

In this post, I will explain how to implement Global Exception Handling in a Spring Boot RESTful service. To engage Spring Boot's Global Exception Handling (@ControllerAdvice or @RestControllerAdvice), you first need to create a class annotated with @ControllerAdvice that will handle exceptions. If you have multiple controllers or HTTP method mappings, this @ControllerAdvice class will be shared across all operations. @RestControllerAdvice is a specialised version of ControllerAdvice annotation to handling methods which are return JSON responses and which is ideal for restful APIs.

I will continue to use the controller example provided in the Spring Boot Key Annotations post. With the @RestControllerAdvice class-level annotation, we need to capture each exception with a method defined with the @ExceptionHandler annotation.

package com.chandana.helloworld.exception;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.MethodArgumentNotValidException;
import org.springframework.web.bind.MissingRequestHeaderException;
import org.springframework.web.bind.MissingServletRequestParameterException;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.servlet.resource.NoResourceFoundException;

/**
 * @author Chandana Napagoda
 */
@RestControllerAdvice
public class GlobalExceptionHandler {

    @ExceptionHandler(MethodArgumentNotValidException.class)
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public ResponseEntity<ErrorResponse> handleValidationExceptions(MethodArgumentNotValidException ex) {
        ErrorResponse errorResponse = new ErrorResponse("Invalid input", ex.getMessage());
        return new ResponseEntity<>(errorResponse, HttpStatus.BAD_REQUEST);
    }

    @ExceptionHandler(MissingServletRequestParameterException.class)
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public ResponseEntity<ErrorResponse> handleMissingParams(MissingServletRequestParameterException ex) {
        ErrorResponse errorResponse = new ErrorResponse("Missing parameter", ex.getParameterName());
        return new ResponseEntity<>(errorResponse, HttpStatus.BAD_REQUEST);
    }

    @ExceptionHandler(MissingRequestHeaderException.class)
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public ResponseEntity<ErrorResponse> handleMissingHeaders(MissingRequestHeaderException ex) {
        ErrorResponse errorResponse = new ErrorResponse("Missing header", ex.getHeaderName());
        return new ResponseEntity<>(errorResponse, HttpStatus.BAD_REQUEST);
    }

    @ExceptionHandler(NoResourceFoundException.class)
    @ResponseStatus(HttpStatus.NOT_FOUND)
    public ResponseEntity<ErrorResponse> handleInvalidControllers(NoResourceFoundException ex) {
        ErrorResponse errorResponse = new ErrorResponse("Resource not found", ex.getResourcePath());
        return new ResponseEntity<>(errorResponse, HttpStatus.BAD_REQUEST);
    }

    @ExceptionHandler(Exception.class)
    @ResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR)
    public ResponseEntity<ErrorResponse> handleAllOtherExceptions(Exception ex) {
        ErrorResponse errorResponse = new ErrorResponse("An error occurred", ex.getMessage());
        return new ResponseEntity<>(errorResponse, HttpStatus.INTERNAL_SERVER_ERROR);
    }
}

@RestControllerAdvice - A class level annotation used to define a global exception handler for entire Spring Boot application. Also it's converts handler response into JSON format.

@ControllerAdvice - A class level annotation used to define a global exception handler for entire Spring Boot application.

@ExceptionHandler - This is a method level annotation used within the @ControllerAdvice class to specify the type of exception that method handles.

Below are a few API responses before and after introducing Controller Advice.

Bad Request Error

Resource Not Found

In addition to your own custom exceptions, Java standard exceptions, database exceptions, security exceptions, and file handling exceptions, these are all the known exception implementations exposed by the Spring Framework.

As a conclusion, by implementing @ControllerAdvice/@RestControllerAdvice in our Spring Boot application, we can easily manage and centralise exception handling across all controllers. Overall this will improve code maintainability and provide better error handling in API level.

Please refrain from copying content to other websites. You can download the Swagger Spring Boot Global Exception Handling Project source code from my GitHub repository.

Spring Boot Key Annotations With Examples

2024-06-26T19:11:00.001+09:30

When developing a RESTful service (REST API) using Spring Boot, need to have clear understanding of some of the annotations provided by Spring Boot Web.

@RestController - RestController is class level annotation used to define a class as a rest full service and it also combines @Controller and @ResponseBody annotations. So the response object is automatically serialised into JSON format.

@RequestMapping - RequestMapping is used to map web requests to a specific controller or method. For RESTful services, the RequestMapping annotation is typically used at the class level, with HTTP method-specific annotations used at the method level.

@GetMapping, @PostMapping, @PutMapping, @DeleteMapping @PatchMapping- These are specialised versions of the @RequestMapping annotation. they are mostly used at the method level to map HTTP methods. These annotations improve code readability and maintainability as well.

@PathVariable - PathVariable annotation is used to extract values from URL templates ( ex: hello/{name})

@RequestParam - RequestParam annotation is used to extract query parameters from the URL

@RequestHeader - RequestHeader annotation is used to extract HTTP request headers from an incoming request

@RequestBody - RequestBody annotation is used to extract the HTTP request body and pass it to the controller method. This can be used with HTTP methods that support a request body (Ex: POST, PUT, Etc)

@ResponseStatus - ResponseStatus annotation is used when we need to pass a different HTTP status other than 200 - OK or map custom exception classes to specific HTTP status codes.

Example Code :

package com.chandana.helloworld;

import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api/greetings")
public class HelloWorldController {

    //GetMapping. Send GET requests to /api/greetings
    @GetMapping
    public String sayHello() {
        return "Hello, World!";
    }

    //Read PathVariable. Send GET requests to /api/greetings/{name}
    @GetMapping("/{name}")
    public String sayHelloToName(@PathVariable String name) {
        return "Hello, " + name + "!";
    }

    //Read RequestBody. Send POST requests to /api/greetings
    @PostMapping
    public String greetWithPost(@RequestBody String name) {
        return "Greetings, " + name + "!";
    }

    // Read RequestParam. Send GET requests to /api/greetings/query?name=ChandanaNapagoda
    @GetMapping("/query")
    public String greetWithRequestParam(@RequestParam String name) {
        return "Hello there, " + name + "!";
    }

    // Read RequestHeader. Send a request to /api/greetings/header
    @GetMapping("/header")
    public String greetWithHeader(@RequestHeader("X-Name") String name) {
        return "Welcome, " + name + "!";
    }

    // Custom HTTP response status for POST requests to /api/greetings/create
    @PostMapping("/create")
    @ResponseStatus(HttpStatus.CREATED)
    public String createGreeting(@RequestBody String name) {
        return "Created greeting for " + name;
    }
}

Please refrain from copying content to other websites

Integrate Swagger 3 with Spring Boot Rest API

2024-06-22T19:47:00.003+09:30

What is Swagger:

Swagger (now known as OpenAPI) is a specification for describing and documenting a REST API. It specifies the format of the REST web services including URL, resources, methods, etc. Swagger generates documentation from the application code and handles the rendering part as well.

In this post, I will integrate Swagger 3 documentation into a Spring Boot 3 based REST web service using Spring Doc OpenAPI. If you need guidance on running or building a Spring Boot project, please refer to my previous post.

Dependencies:

Add the following dependencies to your project POM.xml file. In this project I am using Spring Doc OpenAPI starter and Swagger Core Annotation libraries.

        <dependency>
            <groupId>org.springdoc</groupId>
            <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
            <version>2.5.0</version>
        </dependency>
        <dependency>
            <groupId>io.swagger.core.v3</groupId>
            <artifactId>swagger-annotations</artifactId>
            <version>2.2.22</version>
        </dependency>

Spring Configuration:

To configure the documentation, create a Spring Java Configuration class:

To configure the documentation, you can create a Spring Java Configuration class using Spring’s @Configuration annotation. The following example shows how to generate Swagger documentation based on the RestController classes:

package com.chandana.helloworld.config;

import io.swagger.v3.oas.models.OpenAPI;
import io.swagger.v3.oas.models.info.Contact;
import io.swagger.v3.oas.models.info.Info;
import io.swagger.v3.oas.models.info.License;
import org.springdoc.core.models.GroupedOpenApi;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class OpenApiConfig {

        @Bean
        public OpenAPI customOpenAPI() {
                return new OpenAPI()
                                .info(new Info()
                                                .title("Greetings API Title")
                                                .description("Greetings API Definition")
                                                .version("1.0.0")
                                                .contact(new Contact()
                                                                .name("Chandana Napagoda")
                                                                .url("https://blog.napagoda.com")
                                                                .email("cnapagoda@gmail.com"))
                                                .license(new License()
                                                                .name("Apache 2.0")
                                                                .url("http://www.apache.org/licenses/LICENSE-2.0")));
        }

        @Bean
        public GroupedOpenApi publicApi() {
                return GroupedOpenApi.builder()
                                .group("1.0.0")
                                .pathsToMatch("/api/**")
                                .build();
        }
}

Spring doc OpenAPI uses GroupedOpenApi to configure a subset of the services to be documented and group them by a name, etc. This configuration will include all APIs under /api/** path.

You can provide custom information about your API using the OpenAPI class. This allows you to change default values such as the title, description, version, license, and contact information. With this configuration, the generated documentation will include the specified title, description, version, contact, and license information.

As an alternative to the Spring Java Configuration class, you can define a class with the @OpenAPIDefinition annotation and include complete documentation under that annotation. I have included examples of both in my GitHub repository(spring-boot-3-swagger-3-with-openapidefinition).

Controller and POJO Level Documentation :

To document individual controllers and methods, use annotations from the Swagger library:

@Operation annotation to describe the resources and methods.

@ApiResponses annotation to describe other responses that can be returned by the operation (e.g., 200 OK or 202 Accepted).

@Schema annotation to describe the properties of the POJO (Bean) class or class.

@Parameter annotation to describe the parameters

@Hidden annotation hide specific API endpoint or controller from auto-generated OpenAPI documentation.

package com.chandana.helloworld.controller;

import java.util.HashMap;
import java.util.Map;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.PutMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;

@RestController
@RequestMapping("/api/greetings")
@Tag(name = "greetings", description = "These are greetings API operations")
public class GreetingController {

    private final Map<Integer, GreetingResponse> examples = new HashMap<>();

    @GetMapping("/{id}")
    @Operation(summary = "Get example by ID", description = "Retrieve an example resource by its ID")
    @ApiResponses(value = {
            @ApiResponse(responseCode = "200", description = "Successful operation"),
            @ApiResponse(responseCode = "404", description = "Greeting not found")
    })
    public ResponseEntity<GreetingResponse> getExample(
            @PathVariable @Parameter(allowEmptyValue = false, required = true) int id) {
        GreetingResponse example = examples.get(id);
        if (example == null) {
            return new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }
        return new ResponseEntity<>(example, HttpStatus.OK);
    }

    @PostMapping
    @Operation(summary = "Create a new example", description = "Create a new example resource")
    @ApiResponses(value = {
            @ApiResponse(responseCode = "201", description = "Greeting created"),
            @ApiResponse(responseCode = "400", description = "Invalid input")
    })
    public ResponseEntity<GreetingResponse> createExample(@RequestBody GreetingRequest request) {
        GreetingResponse example = new GreetingResponse("Created " + request.getMessage());
        int id = examples.size() + 1;
        examples.put(id, example);
        return new ResponseEntity<>(example, HttpStatus.CREATED);
    }

    @PutMapping("/{id}")
    @Operation(summary = "Update an example by ID", description = "Update an existing example resource by its ID")
    @ApiResponses(value = {
            @ApiResponse(responseCode = "200", description = "Greeting updated"),
            @ApiResponse(responseCode = "404", description = "Greeting not found")
    })
    public ResponseEntity<GreetingResponse> updateExample(
            @PathVariable @Parameter(allowEmptyValue = false, required = true) int id,
            @RequestBody GreetingRequest request) {
        GreetingResponse example = examples.get(id);
        if (example == null) {
            return new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }
        example.setMessage("Updated " + request.getMessage());
        return new ResponseEntity<>(example, HttpStatus.OK);
    }

    @DeleteMapping("/{id}")
    @Operation(summary = "Delete an example by ID", description = "Delete an example resource by its ID")
    @ApiResponses(value = {
            @ApiResponse(responseCode = "204", description = "Greeting deleted"),
            @ApiResponse(responseCode = "404", description = "Greeting not found")
    })
    public ResponseEntity<Void> deleteExample(
            @PathVariable @Parameter(allowEmptyValue = false, required = true) int id) {
        if (!examples.containsKey(id)) {
            return new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }
        examples.remove(id);
        return new ResponseEntity<>(HttpStatus.NO_CONTENT);
    }
}

After adding these annotations, the final generated Swagger documentation will be enriched with detailed information about your API endpoints and data models.

Swagger documentation published at : http://localhost:8080/swagger-ui/index.html

Greeting Request & Response Model:

package com.chandana.helloworld.controller;

import io.swagger.v3.oas.annotations.media.Schema;

@Schema(description = "The example request message")
public class GreetingRequest {

    @Schema(description = "The example message", example = "Hello, World!")
    private String message;

    // Getters and setters

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

package com.chandana.helloworld.controller;

import io.swagger.v3.oas.annotations.media.Schema;

public class GreetingResponse {

    @Schema(description = "The example response message")
    private String message;

    public GreetingResponse(String message) {
        this.message = message;
    }

    // Getters and setters

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

Please refrain from copying content to other websites. You can download the Swagger Spring Boot Project source code from my GitHub repository.

Building a RESTful Service with Spring Boot 3

2024-06-20T21:02:00.006+09:30

This blog post is an upgraded version of my 2017 guide on building RESTful services using Spring Boot. It includes updates for the latest Spring Boot version, best practices, and enhanced code examples.

Creating Spring boot project is straightforward, You can create the structure of your project using "Spring Initialiser," an online tool where you can add all the desired dependencies to your project's POM file. Since I am a big fan of Maven, I generated a Maven project.

In the Spring Initializer UI, you can choose the Language, Spring Boot Version, Project Group ID, artifact name, etc. Please refer below screenshot for information I have provided while generating the project.

When clicking on "Generate Project", it will download zipped maven project into your computer. Unzip it and import into an IDE. The initial project structure is like below.

The @RestController is a way to implement RESTFul service using Spring. So, this new controller class is going to name as HelloWorldController. @RequestMapping annotation maps HTTP requests to the handler. This @RequestMapping annotation can be used in class-level and/or method-level as well. If you have multiple request mappings for a method or class, you can add one @RequestMapping annotation with a list of values. So my HelloWorldController class looks like below.

package com.chandana.helloworld;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
public class HelloWorldController {
    @RequestMapping("/")
    public String welcome() {// Welcome page, non-rest
        return "Welcome to RestTemplate Example.";
    }

    @RequestMapping("/hello/{name}")
    public Greeting message(@PathVariable String name) {
        Greeting msg = new Greeting(name, "Hello " + name);
        return msg;
    }

    @GetMapping("/greeting/{name}/{message}")
    public Greeting message(@PathVariable String name, @PathVariable String message) {
        Greeting msg = new Greeting(name, message);
        return msg;
    }

}

You can use @GetMapping, @PostMapping, @PutMapping, @DeleteMapping, @PatchMapping annotations as well, these are variation of @RequestMapping to support each HTTP methods.

So everything is in place. you can build and run Spring Boot project using below maven command. It will compile the project and run it.

mvn spring-boot:run

Once the Spring Boot DispatcherServlet is started, you can invoke the following API URLs:

http://localhost:8080/api/hello/Alex - This API supports all HTTP methods.

http://localhost:8080/api/greeting/Alex/WelCome - This API only supports the HTTP GET method.

Final Project Structure:

Greeting POJO class:

package com.chandana.helloworld;

public class Greeting {
    private String player;
    private String message;

    public Greeting(String player, String message) {
        this.player = player;
        this.message = message;
    }

    public String getPlayer() {
        return player;
    }

    public void setPlayer(String player) {
        this.player = player;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

POM XML:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.3.0</version>
        <relativePath/>
        <!-- lookup parent from repository -->
    </parent>
    <groupId>com.chandana</groupId>
    <artifactId>helloworld</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>Hello World</name>
    <description>Demo project for Spring Boot</description>
    <url/>
    <licenses>
        <license/>
    </licenses>
    <developers>
        <developer/>
    </developers>
    <scm>
        <connection/>
        <developerConnection/>
        <tag/>
        <url/>
    </scm>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

HelloWorldController class:

package com.chandana.helloworld;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
public class HelloWorldController {
    @RequestMapping("/")
    public String welcome() {// Welcome page, non-rest
        return "Welcome to RestTemplate Example.";
    }

    @RequestMapping("/hello/{name}")
    public Greeting message(@PathVariable String name) {
        Greeting msg = new Greeting(name, "Hello " + name);
        return msg;
    }

    @GetMapping("/greeting/{name}/{message}")
    public Greeting message(@PathVariable String name, @PathVariable String message) {
        Greeting msg = new Greeting(name, message);
        return msg;
    }

}

Please refrain from copying content to other websites

Custom Mediator cannot be found by synapse-core

2024-04-22T17:48:00.002+09:30

Caused by: java.lang.ClassNotFoundException: com.chandana.test.CustomMediator cannot be found by synapse-core_4.0.0.wso2v20_43
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:512)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:423)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:415)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:155)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at org.apache.synapse.config.xml.ClassMediatorFactory.createSpecificMediator(ClassMediatorFactory.java:100)

If you try to build a WSO2 Custom mediator for Micro Integrator 4.2.0 using Maven, you might encounter above exception in the logs. Based on my understanding, there are two ways to overcome this issue. The first option is to use Integration Studio to export the custom mediator JAR file. However, in my use case, it’s not feasible. After going through the WSO2 Integration Studio source code, I found that it uses different Maven-bundle-plugin instructions than what’s available in the pom.xml file.

I have overcome this issue by adding <_removeheaders>Import-Package,Private-Package</_removeheaders> to the Maven-bundle-plugin instructions.

<plugin>
        <groupId>org.apache.felix</groupId>
        <artifactId>maven-bundle-plugin</artifactId>
        <version>2.3.4</version>
        <extensions>true</extensions>
        <configuration>
          <instructions>
            <Bundle-SymbolicName>CustomClassMediator</Bundle-SymbolicName>
            <Bundle-Name>CustomClassMediator</Bundle-Name>
            <Export-Package>com.chandana.test</Export-Package>
            <DynamicImport-Package>*</DynamicImport-Package>
            <_removeheaders>Import-Package,Private-Package</_removeheaders>
          </instructions>
        </configuration>
      </plugin>

Adding XML Declaration - WSO2

2022-09-10T06:43:00.001+09:30

The XML declaration is a processing instruction that identifies the document is in XML format. While working on one of the integration scenarios, I noticed that WSO2 is not sending the XML declaration in the response or backend request message as expected. However, I was able to see it in the log messages.

To overcome this issue, you can set below the AXIS2 property before sending a message out.

To Add XML declaration:

Also you can use below additional properties to set custom encoding type and set values for standalone.

Integrate With Mutual Certificate Authentication based Service

2020-12-01T08:00:00.017+10:30

There are many ways to secure a Service/API using different Authentication mechanisms. Mutual Certificate authentication(AKA: Client Certificate Authentication) is one of the ways to secure Service or API.

Recently I was trying to integrate with an API which is secured using Mutual Certificate Authentication. In this scenario, the backend service team gave me their certificate-based on PKCS#12.

When integrating this backend system with WSO2 Platform, we have to use 'Multi-HTTPS transport'. With this 'Multi-HTTPS transport', it will grant great flexibility to define different SSL profiles with separate trust stores and key stores for different IPs or hosts. This custom SSL profile can be implemented for Client to Synapse Runtime(Client --> ESB) or Synapse Runtime to Backend(ESB --> Backend). If it is Client to ESB, then you have to define custom SSL Profiles in the transportReceiver section.

In my use-case, it was for ESB to Backend. So I had to add custom SSL profiles into the transportSender section of the axis2.xml file. Since I have a PKCS#12 based certificate file, I don't need to convert it to JKS or anything.

Let's assume Mutual Certificate Authentication based backend API URL is https://backendhost.com/api/users. The Custom SSL profiles configuration in the axis2.xml(or senderprofiles.xml in WSO2 EI 7) is like below.

<transportSender name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLSender">
	<parameter name="non-blocking" locked="false">true</parameter>
	<parameter name="keystore" locked="false">
		<KeyStore>
			<Location>repository/resources/security/wso2carbon.jks</Location>
			<Type>JKS</Type>
			<Password>wso2carbon</Password>
			<KeyPassword>wso2carbon</KeyPassword>
		</KeyStore>
	</parameter>
	<parameter name="truststore" locked="false">
		<TrustStore>
			<Location>repository/resources/security/client-truststore.jks</Location>
			<Type>JKS</Type>
			<Password>wso2carbon</Password>
		</TrustStore>
	</parameter>
	<parameter name="customSSLProfiles">
		<profile>
			<servers>backendhost.com:443</servers>
			<KeyStore>
				<Location>repository/resources/security/backendcertificate.p12</Location>
				<Type>PKCS12</Type>
				<Password>certificatePassword</Password>
				<KeyPassword>certificatePassword</KeyPassword>
			</KeyStore>
		</profile>			
	</parameter>
	<!--<parameter name="HostnameVerifier">DefaultAndLocalhost</parameter>-->
	<!--supports Strict|AllowAll|DefaultAndLocalhost or the default if none specified -->
</transportSender>

Docs:

Multi-HTTPS Transport - WSO2 ESB/EI

Multi-HTTPS Transport - WSO2 EI 7

CSV to SOAP Message conversion with Smooks Mediator

2020-02-20T22:32:00.005+10:30

Smook Mediator available in WSO2 EI/ESB can be used to do various message transformations. For an example read CSV file and transform to XML. I had a requirement where I wanted to load a CSV file content into the database. So, my initial idea was to convert CSV content into XML, then use the XSLT mediator to generate a data service request. But it will be a costly operation with lots of resource consumption.

Therefore, instead of generating a data service request via XSLT, I tried to use the options available in Smooks. Smooks provides FreeMaker based templating support that can be used to convert CSV to SOAP.

Here, first I have created a sample input CSV file and expected Data Service sample request message. With this, it will be very easy to generate data mapping. Below are the examples of CSV and Data Service models

Sample.csv:

UserId,NameFirst,NameLast

1,Chandana,Napagoda

2,John,Tester

SampleOut.xml :

<au:addUsers_batch_req xmlns:au="https://blog.napagoda.com">

<au:addUsers>

<au:UserId>1</au:UserId>

<au:NameFirst>Chandana</au:NameFirst>

<au:NameLast>Napagoda</au:NameLast>

<au:System>-1234</au:System>

</au:addUsers>

<au:addUsers>

<au:UserId>2</au:UserId>

<au:NameFirst>John</au:NameFirst>

<au:NameLast>Tester</au:NameLast>

<au:System>-1234</au:System>

</au:addUsers>

</au:addUsers_batch_req>

When I created the Smook configuration using WSO2 IDE, the view will be like this

Next, click on the input task, then select input CSV file as input data. After that, define fields in the CSV file, using the Input configuration section. In Addition to above, I have skipped the first line, and each record was marked as User, and parent Element defined as Users.

Next, right-click on the 'Input Task', and mouse hover on the 'Add Task' option and click on the 'Apply Template' option.

You will get a dialogue like below to select the output message type. For me, it is XML.

In the next dialogue, I can select my Output file format(Data Service Request).

Once we select the Smooks Output model and click on the 'Finish' button, Initial data mapping screen will look like below.

Next, select 'User' element from the input model and drop it into 'addUsers' output model. This will generate necessary Freemaker template elements to iterate input rows.

Next map each input element into expected output elements. When ding that, you have to select each leaf element from the input model and drop into the output leaf element.

When navigating to source view of Smooks configuration, below configuration has been auto-generated.

<?xml version="1.0" encoding="UTF-8"?><smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd" xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd">

<param name="inputType">input.csv</param>

<param name="input.csv" type="input.type.actived">File:/E:\test\done\Sample.csv</param>

</params>

<csv:reader fields="UserId,NameFirst,NameLast" recordElementName="User" rootElementName="Users" skipLines="1"/>

<ftl:freemarker applyOnElement="#document">

<ftl:template><![CDATA[<?xml version="1.0" encoding="UTF-8"?>

<au:addUsers_batch_req xmlns:au="https://blog.napagoda.com">

<#list .vars["Users"]["User"] as User>

<au:addUsers>

<au:UserId>${.vars["User"]["UserId"]}</au:UserId>

<au:NameFirst>${.vars["User"]["NameFirst"]}</au:NameFirst>

<au:NameLast>${.vars["User"]["NameLast"]}</au:NameLast>

</au:addUsers>

</#list>

</au:addUsers_batch_req>]]></ftl:template>

<param name="modelSrc">File:/E:\test\done\SampleOut.xml</param>

<param name="templateDataProvider">input</param>

</ftl:freemarker>

<resource-config selector="#document">

<resource>org.milyn.delivery.DomModelCreator</resource>

</resource-config>

</smooks-resource-list>

Configure Kerberos based Authentication into WSO2 Datasources

2019-12-03T20:53:00.000+10:30

As you know, Kerberos can be used to authenticate System or users reside on a none secured network. In my use case, I have Windows AD and MSSQL server and I have a couple of WSO2 Servers running on Linux systems.

In my initial setup I have configured, WSO2 servers has to use SQL users. But instead of using SQL users, I prefer to use AD users. So I have started configuring Kerberos and faced a lot of troubles.

If you are starting from scratch, I would recommend writing a Java class to connect MSSQL server using Kerberos. Then You can identify all small mistakes such as character case issues, typos, JDBC driver compatibility issues, etc.

Below are a few issues I have faced:

1) CASE is significant in krb5.conf file

2) I found that we need to use MSSQL JDBC 6.4 version in order to get new tickets. But if you are just planning use pre-generated tickets, you can use the older version as well.

Here are the steps you need follow when configuring Kerberos with WSO2 Enterprise Integrator.

1) Define your krb5.conf File.

In my config file, I have defined two KDC and admin server(KDC) which will handle the database administration requests and all KDC(master and worker) will generate credentials. KDCs are your Active directory server hostnames

[libdefaults]

default_realm = NAPAGODA.COM

[realms]

NAPAGODA.COM = {

kdc = dc1.napagoda.com

kdc = dc2.napagoda.com

admin_server = dc1.napagoda.com

default_domain = NAPAGODA.COM

}

[domain_realm]

.napagoda.com = NAPAGODA.COM

napagoda.com = NAPAGODA.COM

2) . Configure jaas.conf File

This file needs to be created under 'repository/conf/identity/' directory. With WSO2 Enterprise Integrator product they have moved conf directory into product home. However, still, this file needs to be created based on the old folder structure. So make sure to use 'repository/conf/identity/'. Luckily, if not, it will display an error message in the log file(or console).

SQLJDBCDriver {

com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;

};

3). Locating the krb5.conf File

If you have created your krb5.conf file inside lib/security directory of the JRE, you don't need to set the environment variable. Otherwise, you have to pass 'krb5.conf' file location as a 'java.security.krb5.conf' environment variable. Oracle Docs

ex: -Djava.security.krb5.conf=/file/location/krb5.conf

4) Configure WSO2 Datasources

You can use Kerberos based Authentication in your datasources XML files or datasources defined via the 'admin console'. But make sure to pass correct parameters into SQL connection URL to use Kerberos Authentication. Please note, in the connection user and password fields, you can AD user's credentials.

ex: jdbc:sqlserver://your.sql.server.com;database=database_name;integratedSecurity=true;authenticationScheme=JavaKerberos

- Kerberos Debugging in Java
- Microsoft JDBC Driver 6.4 for SQL Server

Compare Two XML Elements and Filter using WSO2 EI

2019-12-01T08:30:00.000+10:30

Think about an integration scenario, where you are getting a list of data from a service endpoint(Data 1) and you want to exclude or match this response against the data returned by another service endpoint(Data 2).

In such a situation, first, you need to have a unique attribute in both data set to exclude or match elements. So in my use case, list of data and matching content is as below

<CodeLists>
 <CodeList>
        <Id>O</Id>
        <Name>Open</Name>
    </CodeList>
    <CodeList>
        <Id>C</Id>
        <Name>Cancelled</Name>
    </CodeList>
    <CodeList>
        <Id>X</Id>
        <Name>Denied</Name>
    </CodeList>
 <CodeList>
        <Id>P</Id>
        <Name>Pending</Name>
    </CodeList>
 <CodeList>
        <Id>D</Id>
        <Name>Duplicate</Name>
    </CodeList>
</CodeLists>

Data 1

<CodeLists>
    <CodeList>
        <Id>O</Id>
        <Name>Open</Name>
    </CodeList>
    <CodeList>
        <Id>P</Id>
        <Name>Pending</Name>
    </CodeList>
</CodeLists>

Data 2

So here I am going to use WSO2 Enterprise Integrator and its XSLT mediator to filter the results. First I am joining all the elements that need to be hidden into a single Synapse property.

<property name="elementsToBeHide" expression="string-join(//CodeLists/CodeList/Id, ',')" scope="default" type="STRING"/>

Then I am calling my XSLT with elementsToBeHide as a parameter.

<xslt key="conf:/xslts/xslt_FilterResults.xslt">
    <property name="elementsToBeHide" expression="$ctx:elementsToBeHide"/>
</xslt>

My XSLT file is stored in the registry. However, if you are using a Micro Integrator profile, you can add this XSLT as a local entry. Inside my XSLT document, I am looping all code list elements and check whether each ID exists in 'elementsToBeHide' parameter. You can use the same XSLT Choose Condition to filter data based on matching or not matching scenarios. In my example, if ID is not matching to the current ID, it will be added to Code element.

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:sy="http://ws.apache.org/ns/synapse" xmlns:fn="http://www.w3.org/2005/xpath-functions"
 xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
 <xsl:strip-space elements="*" />
 
 <xsl:param name="elementsToBeHide" />

 <xsl:template match="/CodeLists">
  <CodeLists>
   <CodeList>
    <xsl:for-each select="CodeList">
                     <xsl:choose>
             <xsl:when test="matches($elementsToBeHide,Id)">
       <!-- match senarios -->
             </xsl:when>
             <xsl:otherwise>
       <!-- not matching -->
              <Code>
       <Name><xsl:value-of select="Name"/></Name>
       <Id><xsl:value-of select="Id"/></Id>
        </Code>
             </xsl:otherwise>
         </xsl:choose>
    </xsl:for-each>
   </CodeList>
  </CodeLists>
 </xsl:template> 
</xsl:stylesheet>

The output of the XSLT transformation is as below and it only includes elements which are not matching to the filtering data set.

<CodeLists>
 <CodeList>
  <Code>
   <Id>C</Id>
   <Name>Cancelled</Name>
  </Code>
  <Code>
   <Id>X</Id>
   <Name>Denied</Name>
  </Code>
  <Code>
   <Id>D</Id>
   <Name>Duplicate</Name>
  </Code>
 </CodeList>
</CodeLists>

XSLT - Modify Date and DateTime value

2019-01-04T22:35:00.001+10:30

If you want to perform date and time related opertions with XSLT 'date' or 'dateTime' values, you can use 'xs:yearMonthDuration' or 'xs:dayTimeDuration' function to achieve this. You can add or reduce years, months, date, hours or even minutes through those functions.

xs:yearMonthDuration

For this function input value is year month duration and it should be passed in ISO_8601 format

n- the [n] is replaced by the value for each of the date and time elements
P - The duration designator(for period).

nY- the number of years.

nM- the number of months.

ex: xs:yearMonthDuration('P1Y2M') - 1 Year and 2 Months

ex: xs:yearMonthDuration('-P1Y2M') - Negative 1 Year and 2 Months

xs:dayTimeDuration

For this function input value is day time duration and it also needs to be passed in ISO_8601 format

n- the [n] is replaced by the value for each of the date and time elements
P - The duration designator(for period).

nD- the number of days.

T- The date and time separator.

nH- the number of hours.

nM- the number of minutes.

nS - the number of seconds.

ex: xs:dayTimeDuration('P3D') - 3 Days

ex: xs:dayTimeDuration('-P3D') - Negative 3 days

ex: xs:dayTimeDuration('P6DT50H80S') - 6 days, 50 hours, and 80 seconds

Example XSLT :

<?xml version="1.0"?>
<xsl:stylesheet version="2.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema" >

<xsl:output method="text"/>

<xsl:variable name="dT" as="xs:dateTime" select="xs:dateTime('2016-02-29T00:00:01')"/>
<xsl:variable name="d" as="xs:date" select="xs:date('2016-02-29')"/>

<xsl:template match="/">
<xsl:text>Add 1 Year and 2 Months : </xsl:text> <xsl:value-of select="($d+ xs:yearMonthDuration('P1Y2M'))"/>
<xsl:text>
Negative 1 Year and 2 Month : </xsl:text> <xsl:value-of select="($d+ xs:yearMonthDuration('-P1Y2M'))"/>

<xsl:text>
Add 3 Days : </xsl:text> <xsl:value-of select="($dT+ xs:dayTimeDuration('P3D'))"/>
<xsl:text>
Negative 3 days : </xsl:text> <xsl:value-of select="($dT+xs:dayTimeDuration('-P3D'))"/>
<xsl:text>
Add 6 days, 50 hours, and 80 seconds : </xsl:text> <xsl:value-of select="($dT+ xs:dayTimeDuration('P6DT50H80S'))"/>
</xsl:template>
</xsl:stylesheet>

WSO2 EI/ESB Change Backend Response Status Code

2018-07-26T19:24:00.005+09:30

With WSO2 Enterprise Integrator or Enterprise Service Bus servers, if you want to change the backend server response code, you can set custom status status code as below:

If you are getting 'HTTP/1.1 202 Accepted' response from your backend, you need to set "SC_ACCEPTED" as false by setting a custom SC_ACCEPTED

Ex: Receiving 202 from backend and change it to 200

<then>

</then>

<else/>

</filter>

Read XML content of the LocalEntry in WSO2 ESB

2018-06-06T17:38:00.000+09:30

Local Entry is one of the data storage points in the WSO2 Enterprise Service Bus, where you can store text strings, XML content, and File URLs. I have been asked a question that, how can we read the content of the local entry if it as an XML file.

Below is an example to read the content of the local entry(Name: NameOfLocalEntry) and assign the value of XML elements as properties.

Content of Local Entry:

<?xml version="1.0"?>

<userName>chandana</userName>

<URL>blog.napagoda.com</URL>

</Server>

Synapse Configuration:

Payload Factory Mediators for JSON Message

2018-04-30T21:05:00.000+09:30

WSO2 ESB/EI PayloadFactory mediator can be used to replace the contents of a message. As an example, if the backend is accepting a different message structure than what you are receiving, you can use PayloadFactory mediator to change the message structure. Other than that, if you want to transform XML payload into JSON or JSON to XML with different message structure you can use this mediator.

Recently, I faced an issue while transforming a message into JSON which is originally read from a CSV file(using Smooks Mediator). This message can have empty values in the content sometimes and such messages look like below:

<Users>
<User>
<FirstName>Chandana</FirstName>
<MiddleName/>
</Users>
</User>

I initially wrote a payload factory mediator like below:

<payloadFactory media-type="json">
<format>
{
"UserInfo": {
"fName": "$1",
"mName": "$2"

}
}
</format>
<args>
<arg evaluator="xml" expression="//User/FirstName"/>
<arg evaluator="xml" expression="//User/MiddleName"/>
</args>
</payloadFactory>

JSON Output from the Payloadfactory mediator :

{
"UserInfo": {
"fName": "Chandana",
"mName": "{"MiddleName":null}",

}
}

So you can see that JSON output is malformed, but this will work perfectly when you have a value for MiddleName field.

To avoid this issue, what we have to do is, when we are passing arguments to the PayloadFactory mediator, we should pass text values(ex: FirstName/text()) instead of the XML object.

<payloadFactory media-type="json">
<format>
{
"UserInfo": {
"fName": "$1",
"mName": "$2"

}
}
</format>
<args>
<arg evaluator="xml" expression="//User/FirstName/text()"/>
<arg evaluator="xml" expression="//User/MiddleName/text()"/>
</args>
</payloadFactory>

WSO2 ESB - How to use filter inside iterate mediator

2018-02-22T06:58:00.000+10:30

WSO2 ESB's Iterate mediator plays a very powerful role in the Splitter Enterprise Integration Pattern. Splitter Enterprise Integration Pattern is used when messages contain multiple elements that might have to be processed in different ways. The Splitter breaks out the composite message into a series of individual messages, each containing data related to one item.

WSO2 ESB Iterate mediator split the message based on a given expression and process them separately. So think about a use case like you are getting multiple order items and you want to enrich each order item, by calling another endpoint and finally we need to aggregate all the enriched items.

There you can use the Iterate mediator and after that aggregate mediator to aggregate all the enriched items.

Example:

ESB Proxy Service:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="IteService"
       transports="http,https"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <iterate id="aggregateid"
                  preservePayload="true"
                  attachPath="//Collection"
                  expression="//Collection/Job"
                  sequential="true">
            <target>
               <sequence>
                  <enrich>
                     <source type="body" clone="true"/>
                     <target type="property" property="originalRequest"/>
                  </enrich>
                  <payloadFactory media-type="xml">
                     <format>
                        <p:echoInt xmlns:p="http://echo.services.core.carbon.wso2.org">
                           <in>$1</in>
                        </p:echoInt>
                     </format>
                     <args>
                        <arg evaluator="xml" expression="$body/Collection/Job/Id"/>
                     </args>
                  </payloadFactory>
                  <property name="SOAPAction"
                            value="&#34;echoInt&#34;"
                            scope="transport"
                            type="STRING"/>
                  <call>
                     <endpoint>
                        <address uri="https://localhost:8243/services/echo"/>
                     </endpoint>
                  </call>
                  <property xmlns:ns="http://echo.services.core.carbon.wso2.org"
                            name="echoResponse"
                            expression="$body/ns:echoIntResponse"
                            scope="default"
                            type="OM"/>
                  <enrich>
                     <source type="property" clone="true" property="originalRequest"/>
                     <target type="body"/>
                  </enrich>
                  <enrich>
                     <source type="property" clone="true" property="echoResponse"/>
                     <target action="child" xpath="$body/Collection"/>
                  </enrich>
               </sequence>
            </target>
         </iterate>
         <property name="Collections" scope="default">
            <Collections xmlns=""/>
         </property>
         <aggregate id="aggregateid">
            <completeCondition>
               <messageCount min="-1" max="-1"/>
            </completeCondition>
            <onComplete expression="$body/Collection" enclosingElementProperty="Collections">
               <respond/>
            </onComplete>
         </aggregate>
      </inSequence>
   </target>
   <description/>
</proxy>

Request:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <Collection>
         <Job>
            <Id>12345</Id>
         </Job>
         <Job>
            <Id>12346</Id>
         </Job>
      </Collection>
   </soapenv:Body>
</soapenv:Envelope>

Let me expand this use case. Assume we just need to enrich some of the items only. To do that, we have to use filter mediator inside the Iterate mediator. If we update our existing code with the filter mediator logic, it will never reach to the aggregate on-complete logic. The reason is aggregate mediator waits for total messages that came into the iterate mediator, but due to the filter logic, some messages aren't supposed to be included in the aggregation. To avoid this, what we can do is, we have to define this Iterate mediator inside the In-Sequence and after all the logic inside the Iterate mediator( before the tag) you have to add a loopback mediator and need to move aggregate mediator into Out-Sequence. This will call the aggregate mediator for all the messages.

Example:

ESB Proxy Service:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="IteFilterService"
       transports="http,https"
       statistics="disable"
       trace="disable"
       startOnLoad="true">
   <target>
      <inSequence>
         <iterate id="aggregateid"
                  preservePayload="true"
                  attachPath="//Collection"
                  expression="//Collection/Job"
                  sequential="true">
            <target>
               <sequence>
                  <property name="JobId"
                            expression="string(//Job/Id)"
                            scope="default"
                            type="STRING"/>
                  <filter source="$ctx:JobId" regex="[0-9]+">
                     <log level="full">
                        <property name="JobId" expression="string(//Job/Id)"/>
                     </log>
                     <enrich>
                        <source type="body" clone="true"/>
                        <target type="property" property="originalRequest"/>
                     </enrich>
                     <payloadFactory media-type="xml">
                        <format>
                           <p:echoInt xmlns:p="http://echo.services.core.carbon.wso2.org">
                              <in>$1</in>
                           </p:echoInt>
                        </format>
                        <args>
                           <arg evaluator="xml" expression="$ctx:JobId"/>
                        </args>
                     </payloadFactory>
                     <property name="SOAPAction"
                               value="&#34;echoInt&#34;"
                               scope="transport"
                               type="STRING"/>
                     <call>
                        <endpoint>
                           <address uri="https://localhost:8243/services/echo"/>
                        </endpoint>
                     </call>
                     <property xmlns:ns="http://echo.services.core.carbon.wso2.org"
                               name="echoResponse"
                               expression="$body/ns:echoIntResponse"
                               scope="default"
                               type="OM"/>
                     <enrich>
                        <source type="property" clone="true" property="originalRequest"/>
                        <target type="body"/>
                     </enrich>
                     <enrich>
                        <source type="property" clone="true" property="echoResponse"/>
                        <target action="child" xpath="$body/Collection"/>
                     </enrich>
                  </filter>
                  <loopback/>
               </sequence>
            </target>
         </iterate>
      </inSequence>
      <outSequence>
         <property name="Collections" scope="default">
            <Collections xmlns=""/>
         </property>
         <aggregate id="aggregateid">
            <completeCondition>
               <messageCount min="-1" max="-1"/>
            </completeCondition>
            <onComplete expression="$body/Collection" enclosingElementProperty="Collections">
               <respond/>
            </onComplete>
         </aggregate>
      </outSequence>
   </target>
   <description/>
</proxy>

Request:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <Collection>
         <Job>
            <Id>AAAA</Id>
         </Job>
         <Job>
            <Id>12345</Id>
         </Job>
         <Job>
            <Id>12346</Id>
         </Job>
      </Collection>
   </soapenv:Body>
</soapenv:Envelope>

WSO2 ESB/EI Callout Mediator Error Scenario

2017-12-19T08:56:00.000+10:30

When using WSO2 Enterprise Service Bus, you can use Call, Send and Callout mediators to invoke a service. If you are calling multiple service calls within your meditation sequence, you have to use either Call mediator or Callout mediator.

As per the documentation, WSO2 Team is recommending to use Call mediator instead of the Callout mediator, due to much better performance. However, due to some legacy requirements, we might need to stay with Callout mediator for the time being.

In my use case, there are some mediation scenarios with mutual SSL. So if you have noticed an "UnrecoverableKeyException: Password verification failed" exception in the WSO2Carbon log file and terminal when invoking an endpoint(backend service) using callout mediator, I would recommend you to check the Java SSL keyStore Password(values of the javax.net.ssl.keyStorePassword and javax.net.ssl.trustStorePassword environment variables) in the /bin/wso2server.sh file or relevant location.

TID: [-1234] [] [2017-12-16 13:04:28,193]  INFO {org.apache.axis2.transport.http.HTTPSender} -  Unable to sendViaPost to url[http://blog.napagoda.com] {org.apache.axis2.transport.http.HTTPSender} UUID:[d7602d4a-17ac-4b95-b459-7fec3ffa44c0]
java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
        at javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:248)
        at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:285)
        at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:155)
        at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
        at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:688)
        at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:195)
        at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
        at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
        at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
        at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
        at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
        at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
        at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
        at org.apache.synapse.message.senders.blocking.BlockingMsgSender.sendReceive(BlockingMsgSender.java:310)
        at org.apache.synapse.message.senders.blocking.BlockingMsgSender.send(BlockingMsgSender.java:216)
        at org.apache.synapse.mediators.builtin.CalloutMediator.mediate(CalloutMediator.java:180)
        at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:89)
        at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:56)
        at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149)
        at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:221)
        at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:89)
        at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:56)
        at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149)
        at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:197)
        at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
        at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)
        at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)
        at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
        at java.security.Provider$Service.newInstance(Provider.java:1617)
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
        at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
        at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
        at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122)
        ... 33 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:852)
        at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:732)
        at sun.reflect.GeneratedConstructorAccessor131.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.security.Provider$Service.newInstance(Provider.java:1595)
        ... 38 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
        ... 48 more

WSO2 ESB/EI Send XML content to backend

2017-12-10T10:05:00.000+10:30

When sending XML content inside the payload to the backend via WSO2 ESB, we have to encode it and send it. In my usecase, I have a Data Service which is accepting XML content as a parameter.

To implement this requirement, we can't directly define CDATA(blocks of text that are not parsed by XML parser) inside the payload factory mediator. So we have two option to do so.

The first option is that Encode XML content using Script mediator and use encoded value inside the payload factory mediator. You can read Hasitha's blog on this topic.

The second option is storing the format section of the payload factory mediator, in the registry. There you can directly define the CDATA tags inside the XML content stored in the registry. This allows you to define CDATA inside payload factory mediator.

An example usecase is as below:

PayloadFacroy Mediator:

<?xml version="1.0" encoding="UTF-8"?>
<payloadFactory media-type="xml">
   <format key="conf:/format/Common_Error_Message.xml" />
   <args>
      <arg evaluator="xml" expression="$ctx:ORIGINAL_MESSAGE" />
      <arg expression="$axis2:HTTP_SC" />
   </args>
</payloadFactory>

Registry Content:

<?xml version="1.0" encoding="UTF-8"?>
<xs:logError xmlns:xs="http://wso2.org/services/data/error">
   <xs:originalPayload><![CDATA[$1]]></xs:originalPayload>
   <xs:statusCode>$2</xs:statusCode>
</xs:logError>

Secure Spring Boot REST API using Basic Authentication

2017-10-14T00:30:00.000+10:30

This is the third post of my Spring Boot Blog post series. In the very first post, I talked about my experience with creating RESTFul Services using Spring Boot. Then I have expanded the sample to integrate with Swagger documentation. In this post, I am going to expand above sample with security aspect.

What is API Security

API Security is a wide area with many different definitions, meanings, and solutions. The main key terms in API security are Authorization, Authentication, Encryption, Federation, and Delegation. However, I am not going to talk about each of them here.

What is Authentication

Authentication is used to reliably determine the identity of an end user and give access to the resources based on the correctly identified user.

What is Basic Authentication

Basic Authentication is the simplest way to enforce access controling to resources. Here, the HTTP user agent provides the username and the password when making a request. The string containing the username and password separated by a colon is Base64 encoded before sending to the backend when authentication is required.

How to Invoke Basic Auth Protected API

Option 1: Send Authorization header. This value is base64 encoded username:password

Ex: "Authorization: Basic Y2hhbmRhbmE6Y2hhbmRhbmE="

curl -X GET http://localhost:8080/admin/hello/chandana -H 'authorization: Basic Y2hhbmRhbmE6Y2hhbmRhbmE='

Option 2: Using URL:

curl -X GET -u username:password http://localhost:8080/admin/hello/chandana

OK, we talked about basic stuff. So let's move to see how to secure a REST API using Spring Security. You can download the initial sample code from my GitHub repo(Swagger Spring Boot Project source code)

To enhance our previous sample with basic auth security, first I am going to add "spring-boot-starter-security" and "spring-boot-starter-tomcat" dependencies into the pom file.

        <!-- -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-security</artifactId>
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.1.0</version>
        </dependency>

Next step is that our configuration class is annotated with @EnableWebSecurity annotation and configuration class is extended from the WebSecurityConfigurerAdapter. The EnableWebSecurity annotation will enable Spring-Security web security support.

@Configuration
@EnableSwagger2
@EnableWebSecurity
public class ApplicationConfig extends WebSecurityConfigurerAdapter {

Overridden configure(HttpSecurity) method is used to define which URL paths should be secured and which should not be. In my example "/" and "/api" paths are not required any authentication and any other paths(ex: "admin") should be authenticated with basic auth.

@Override
protected void configure(HttpSecurity http) throws Exception {
        http.csrf().disable();
        http.authorizeRequests().antMatchers("/", "/api/**").permitAll()
        .anyRequest().authenticated();
        http.httpBasic().authenticationEntryPoint(basicAuthenticationPoint);
}

In the configureGlobal(AuthenticationManagerBuilder) method, I have created an in-memory user store with a user called 'chandana'. There I have added username, password, and userole for the in-memory user.

    @Autowired
    public void configureGlobal(AuthenticationManagerBuilder auth) throws Exception {
        auth.inMemoryAuthentication().withUser("chandana").password("chandana").roles("USER");
    }

In Addition to that, you can see that I have added autowired BasicAuthenticationPoint, into my config class. Purpose of the BasicAuthenticationEntryPoint class is to set the "WWW-Authenticate" header to the response. So, web browsers will display a dialog to enter usename and password based on basic authentication mechanism(WWW-Authenticate header)

Then you can run the sample using "mvn spring-boot:run". When you are accessing "localhost:8080/api/hello/chandana" basic authentication is not required to invoke the api. However, if you try to access the "localhost:8080/admin/hello/chandana" it will be required to provide basic auth credentials to access the resource.

AppConfig class:

 package com.chandana.helloworld.config;  
 import org.springframework.beans.factory.annotation.Autowired;  
 import org.springframework.context.annotation.Bean;  
 import org.springframework.context.annotation.Configuration;  
 import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;  
 import org.springframework.security.config.annotation.web.builders.HttpSecurity;  
 import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;  
 import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;  
 import springfox.documentation.builders.ApiInfoBuilder;  
 import springfox.documentation.builders.PathSelectors;  
 import springfox.documentation.builders.RequestHandlerSelectors;  
 import springfox.documentation.service.ApiInfo;  
 import springfox.documentation.service.Contact;  
 import springfox.documentation.spi.DocumentationType;  
 import springfox.documentation.spring.web.plugins.Docket;  
 import springfox.documentation.swagger2.annotations.EnableSwagger2;  
 @Configuration  
 @EnableSwagger2  
 @EnableWebSecurity  
 public class ApplicationConfig extends WebSecurityConfigurerAdapter {  
   @Autowired  
   private BasicAuthenticationPoint basicAuthenticationPoint;  
   @Bean  
   public Docket api() {  
     return new Docket(DocumentationType.SWAGGER_2)  
         .apiInfo(getApiInfo())  
         .select()  
         .apis(RequestHandlerSelectors.basePackage("com.chandana.helloworld.controllers"))  
         .paths(PathSelectors.any())  
         .build();  
   }  
   @Override  
   protected void configure(HttpSecurity http) throws Exception {  
     http.csrf().disable();  
     http.authorizeRequests().antMatchers("/", "/api/**").permitAll()  
     .anyRequest().authenticated();  
     http.httpBasic().authenticationEntryPoint(basicAuthenticationPoint);  
   }  
   private ApiInfo getApiInfo() {  
     Contact contact = new Contact("Chandana Napagoda", "http://blog.napagoda.com", "cnapagoda@gmail.com");  
     return new ApiInfoBuilder()  
         .title("Example Api Title")  
         .description("Example Api Definition")  
         .version("1.0.0")  
         .license("Apache 2.0")  
         .licenseUrl("http://www.apache.org/licenses/LICENSE-2.0")  
         .contact(contact)  
         .build();  
   }  
   @Autowired  
   public void configureGlobal(AuthenticationManagerBuilder auth) throws Exception {  
     auth.inMemoryAuthentication().withUser("chandana").password("chandana").roles("USER");  
   }  
 }

BasicAuthenticationEntryPoint class:

 package com.chandana.helloworld.config;  
 import org.springframework.security.core.AuthenticationException;  
 import org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint;  
 import org.springframework.stereotype.Component;  
 import java.io.IOException;  
 import java.io.PrintWriter;  
 import javax.servlet.ServletException;  
 import javax.servlet.http.HttpServletRequest;  
 import javax.servlet.http.HttpServletResponse;  
 @Component  
 public class BasicAuthenticationPoint extends BasicAuthenticationEntryPoint {  
   @Override  
   public void commence(HttpServletRequest request, HttpServletResponse response, AuthenticationException authEx)  
       throws IOException, ServletException {  
     response.addHeader("WWW-Authenticate", "Basic realm=" +getRealmName());  
     response.setStatus(HttpServletResponse.SC_UNAUTHORIZED);  
     PrintWriter writer = response.getWriter();  
     writer.println("HTTP Status 401 - " + authEx.getMessage());  
   }  
   @Override  
   public void afterPropertiesSet() throws Exception {  
     setRealmName("Chandana");  
     super.afterPropertiesSet();  
   }  
 }

You can download Spring Boot Basic Auth Project source code from my GitHub repo as well.

Introduction to WSO2 Registry Mounting

2017-09-16T22:53:00.000+09:30

This post is based on the common questions raised about registry mounting and how it works etc. Below are the main questions people ask:

1). How mounting works?
2). What is the difference between Config Registry and Governance Registry?
3). Can I use databases other than H2 for Local Registry?
4). What is meant by mount path and target path?
5). Do I need to configure “remoteInstance” URL?
6). What should I use as the cacheId?

So let's start with how to configure a registry mount. When you are configuring the registry mount, you have to add the relevant data source to the master-datasources.xml file. In addition to that, you have to add mounting related configuration into the registry.xml file as well.

In the master-datasources.xml file you have to just configure a JDBC data source by providing JDBC URL, username, password, validation queries, connection optimization parameters, etc. An example data source entry will look like below. Note: You need to add below configuration in addition to the existing 'WSO2CarbonDB' data source.

        <datasource>
            <name>WSO2CarbonDB_Gov</name>
            <description>The datasource used for registry- config/governance</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB_Gov</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://blog.napagoda.com:3306/REGISTRY_DB?autoReconnect=true</url>
                    <username>chandana</username>
                    <password>password</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

In the registry.xml file, there are many vexed areas available. So let’s see an example mounting configuration first.

<dbConfig name="mounted_registry">
        <dataSource>jdbc/WSO2CarbonDB_Gov</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
        <id>instanceid</id>
        <dbConfig>mounted_registry</dbConfig>
        <readOnly>false</readOnly>
        <enableCache>true</enableCache>
        <registryRoot>/</registryRoot>
        <cacheId>chandana@jdbc:mysql://blog.napagoda.com:3306/REGISTRY_DB?autoReconnect=true</cacheId>
</remoteInstance>

<mount path="/_system/config" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/apimconfig</targetPath>
</mount>
<mount path="/_system/governance" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/governance</targetPath>
</mount>

You can see that when defining a mounting configuration, I have added four sections of configurations. They are ‘dbConfig’, ‘remoteInstance’ and two sections of the ‘mount’ entry.

I think it's easy to explain from the mount entry first, then remoteInstance and dbConfig. In the mount entry, you can configure path, overwrite, targetPath, and instanceId.

Mount

path - Path is a location in the registry which is similar to a file system path. Resources stored inside this path will be store in relevant configured DB.
overwrite- (Virtual, True, False)Whether an existing collection/resource at the given path would be overwritten or not. Virtual means changes are only stored in the memory and will not be written into the DB.
instanceId - Reference to the ‘remoteInstance’.
targetPath - The path which is stored in the database.

In nutshell, any registry paths which are starting with the value in the path section will be stored in the DB against targetPath(path will be replaced with targetPath and stored in the DB). When retrieving registry path it will do the reverse replacement as well. So this target path will not be visible to you at all. If you are too curious to know about that, you can verify it by querying REG_PATH table.

remoteInstance

'remoteInstance' is the mapping between 'dbConfig' and Mounts. This mapping is handled via 'id' and 'dbConfig' elements. The 'id' value referred in the each mounting configurations and value of dbConfig element should be same as dbConfig name. In addition to that 'cacheId' is one of the most important configurations in this section.

url - Registry URL of the local registry instance. This is only used in the WSO2 Governance Registry product. So you can use any value for the other products.
readOnly - Whether the instance is read-only.
registryRoot - The root of the registry instance.
enableCache - Whether caching is enabled or not.
cacheId - This is an unique identification of the remote instance used in distributed caching layer. Here we are recommending to use the cache id as registry DBUsername@DBUrl.

dbConfig

This dbConfig is a reference to the data source added in the master-datasources.xml file. Note that you should not remove or modify default dbConfig available in the registry.xml file. Instead of that, you need to add a new dbConfig element. Further, as the name of the newly adding dbConfig, you should use a name other than 'wso2registry', since it has been used as the default dbConfig name.

So, let me answer to other questions. Any WSO2 product(released before 2018) internally consist three registry spaces. they are local, config and governance.

Local Registry(repository) is used to store instance specific information such as last index time', etc.
Config Registry(repository) is the place to store information that can only be shared with same products and if multinode product cluster, this section will be shared.
Governance Registry(repository) is the place to store configurations and data that are shared across the whole WSO2 platform.

We are recommending to store config and governance sections in an external database system. Since the Local Registry(repository) section is instance specific, we are recommending to store it with default H2 database. Information which are stored in the local registry are fail-safe and can be recovered. Please note that if you are willing to store Local section in an external RDBMS, you have to create a separate database(schema) for each instance.

So let’s move on to validating my mounting configuration. In your 'remoteInstance' configuration you have to correctly refer the dbConfig name. This DB config name should not be the same one which we used for Local Registry. In addition that, you have to properly map each 'mount' section to the 'remoteInstance' using the instanceId.

If you have any questions which are related to registry mounting, you can comment here. I am happy to help you.

Integrating Swagger with Spring Boot REST API

2017-09-12T00:04:00.001+09:30

Update Notice: This blog post was published on 2017 and may contain outdated information. For the most up-to-date guide on integrating Swagger with Spring Boot REST API, please refer to my latest post : https://blog.napagoda.com/2024/06/integrate-swagger-3-with-spring-boot3-api.html

In the last post, I talked about my experience with creating RESTFul Services using Spring Boot. When creating a REST API, proper documentation is a mandatory part of it.

What is Swagger?

Swagger(Swagger 2) is a specification for describing and documenting a REST API. It specifies the format of the REST web services including URL, Resources, methods, etc. Swagger will generate documentation from the application code and handle the rendering part as well.

In this post, I am going to integrate Swagger 2 documentation into a Spring Boot based REST web service. So I am going to use Springfox implementation to generate the swagger documentation. If you want to know how to run/build Spring Boot project, please refer my previous post.

Springfox provides two dependencies to generate API Doc and Swagger UI. If you are not expecting to integrate Swagger UI into your API level, no need to add Swagger UI dependency.

<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>2.7.0</version>
</dependency>

<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>2.7.0</version>
</dependency>

@EnableSwagger2 annotation enables Springfox Swagger support in the class. To document the service, Springfox uses a Docket. The Docket helps to configure a subset of the services to be documented and group them by a name, etc. The most hidden concept is that the Springfox works by examining an application at runtime using API semantics based on spring configurations. In other words, you have to create a Spring Java Configuration class which uses spring’s @Configuration

In My example, I am generating a swagger documentation based on the RestController classes I have added.


import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;

@Configuration
@EnableSwagger2
public class ApplicationConfig {

    @Bean
    public Docket api() {
        return new Docket(DocumentationType.SWAGGER_2)
                .select()
                .apis(RequestHandlerSelectors.basePackage("com.chandana.helloworld.controllers"))
                .paths(PathSelectors.any())
                .build();
    }
}

Since I have added two controllers, this will group(tag) each controller related APIs separately.

Out of the box, Springfox provides five predicates and they are any, none, withClassAnnotation, withMethodAnnotation and basePackage.

ApiInfo

Swagger provides some default values such as “API Documentation”, “Created by Contact Email”, “Apache 2.0”. So you can change these default values by adding apiInfo(ApiInfo apiInfo) method. The ApiInfo class contains custom information about the API.

@Bean
    public Docket api() {
        return new Docket(DocumentationType.SWAGGER_2)
                .apiInfo(getApiInfo())
                .select()
                .apis(RequestHandlerSelectors.basePackage("com.chandana.helloworld.controllers"))
                .paths(PathSelectors.any())
                .build();
    }

    private ApiInfo getApiInfo() {
        Contact contact = new Contact("Chandana Napagoda", "http://blog.napagoda.com", "cnapagoda@gmail.com");
        return new ApiInfoBuilder()
                .title("Example Api Title")
                .description("Example Api Definition")
                .version("1.0.0")
                .license("Apache 2.0")
                .licenseUrl("http://www.apache.org/licenses/LICENSE-2.0")
                .contact(contact)
                .build();
    }

Once ApiInfo is added, the generated documentation looks similar to this:

Controller and POJO Level Documentation

@Api annotation is used to explain each rest controller class.
@ApiOperation annotation is used to explain to describe the resources and methods.
@ApiResponse annotation is used to explain to describe other responses that can be returned by the operation.ex: 200 ok or 202 accepted, etc.
@ApiModelProperty annotation to describe the properties of the POJO(Bean) class.

After adding above annotation, final generated swagger documentation looks like below:

Spring RestController class:


package com.chandana.helloworld.controllers;

import com.chandana.helloworld.bean.Greeting;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.ApiResponse;
import io.swagger.annotations.ApiResponses;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
@Api(value = "user", description = "Rest API for user operations", tags = "User API")
public class HelloWorldController {

    @RequestMapping(value = "/hello/{name}", method = RequestMethod.GET, produces = "application/json")
    @ApiOperation(value = "Display greeting message to non-admin user", response = Greeting.class)
    @ApiResponses(value = {
            @ApiResponse(code = 200, message = "OK"),
            @ApiResponse(code = 404, message = "The resource not found")
    }
    )
    public Greeting message(@PathVariable String name) {
        Greeting msg = new Greeting(name, "Hello " + name);
        return msg;
    }
}

Greeting model class:


package com.chandana.helloworld.bean;

import io.swagger.annotations.ApiModelProperty;

public class Greeting {

    @ApiModelProperty(notes = "Provided user name", required =true)
    private String player;

    @ApiModelProperty(notes = "The system generated greeting message" , readOnly =true)
    private String message;

    public Greeting(String player, String message) {
        this.player = player;
        this.message = message;
    }

    public String getPlayer() {
        return player;
    }

    public void setPlayer(String player) {
        this.player = player;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

AppConfig class:


package com.chandana.helloworld.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import springfox.documentation.builders.ApiInfoBuilder;
import springfox.documentation.builders.PathSelectors;
import springfox.documentation.builders.RequestHandlerSelectors;
import springfox.documentation.service.ApiInfo;
import springfox.documentation.service.Contact;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;

@Configuration
@EnableSwagger2
public class ApplicationConfig {

    @Bean
    public Docket api() {
        return new Docket(DocumentationType.SWAGGER_2)
                .apiInfo(getApiInfo())
                .select()
                .apis(RequestHandlerSelectors.basePackage("com.chandana.helloworld.controllers"))
                .paths(PathSelectors.any())
                .build();
    }

    private ApiInfo getApiInfo() {
        Contact contact = new Contact("Chandana Napagoda", "http://blog.napagoda.com", "cnapagoda@gmail.com");
        return new ApiInfoBuilder()
                .title("Example Api Title")
                .description("Example Api Definition")
                .version("1.0.0")
                .license("Apache 2.0")
                .licenseUrl("http://www.apache.org/licenses/LICENSE-2.0")
                .contact(contact)
                .build();
    }
}

You can download Swagger Spring Boot Project source code from my GitHub repo as well.

Building a RESTFul Service using Spring Boot

2017-09-08T02:37:00.004+09:30

Update Notice: This blog post was published on 2017 and may contain outdated information. For the most up-to-date guide on developing RESTful services with Spring Boot, please refer to my latest post: https://blog.napagoda.com/2024/06/building-restful-service-spring-boot-3.html Thank you!

Everyone is talking about Microservices such as WSO2 Microservice Framework, Spring Boot, etc. Since I haven't worked on any Spring related project since a very long time, I thought to implement a simple RESTFul service using Spring Boot.

So I started with Spring documentation. It is straightforward. You can create the structure of your project using "Spring Initializr". This is an online tool where you can add all the desired dependencies to your project POM file. Since I am a big fan of Maven, I am generating a maven project.

In the Spring Initializr UI, you can choose the Language, Spring Boot Version, Project Group ID, artifact name, etc. Please refer below screenshot for information I have provided while generating the project.

When clicking on "Generate Project", it will download zipped maven project into your computer. Unzip it and import into an IDE. The initial project structure is like below.

In my HelloWorld REST service implementation, it accepts user's name as a path parameter(or URL parameter) and returns a greeting JSON payload(response). So I am expecting to invoke my REST service by calling below URL: APP_NAME/api/hello/chandana.

The @RestController is a way to implement RESTFul service using Spring. So, this new controller class is going to name as HelloWorldController. @RequestMapping annotation maps HTTP requests to the handler. This @RequestMapping annotation can be used in class-level and/or method-level as well. If you have multiple request mappings for a method or class, you can add one @RequestMapping annotation with a list of values. So my HelloWorldController class looks like below.

package com.chandana.helloworld;

import com.chandana.helloworld.bean.Greeting;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
public class HelloWorldController {

    @RequestMapping("/")
    public String welcome() {//Welcome page, non-rest
        return "Welcome to RestTemplate Example.";
    }

    @RequestMapping("/hello/{name}")
    public Greeting message(@PathVariable String name) {

        Greeting  msg = new Greeting(name, "Hello " + name);
        return msg;
    }

}

Note: If you notice Spring Boot 1.5.6 not importing classes correctly and displaying an error message as "Cannot resolve symbol RestController" in your IDE, you need to downgrade the spring version that is used in the project. Spring Boot 1.5.6 by default uses Spring 4.3.10.RELEASE dependency and it need to be downgraded to 4.3.9.RELEASE. So please add <spring.version>4.3.9.RELEASE</spring.version> on the properties section of your POM file.

So everything is in place. I can build and run Spring Boot project using below maven command. It will compile the project and run it.

mvn spring-boot:run

While starting the server you can notice registered REST service URL in the console like below

INFO 9556 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/hello/{name}]}" onto public com.chandana.helloworld.bean.Greeting com.chandana.helloworld.HelloWorldController.message(java.lang.String)

INFO 9556 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/]}" onto public java.lang.String com.chandana.helloworld.HelloWorldController.welcome()2017-0
Finally, Can invoke REST Service by accessing this URL: http://localhost:8080/api/hello/NAME

Final Project Structure:

Greeting POJO class:

package com.chandana.helloworld.bean;

public class Greeting {

    private String player;
    private String message;

    public Greeting(String player, String message) {
        this.player = player;
        this.message = message;
    }

    public String getPlayer() {
        return player;
    }

    public void setPlayer(String player) {
        this.player = player;
    }

    public String getMessage() {
        return message;
    }

    public void setMessage(String message) {
        this.message = message;
    }
}

POM XML:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>com.chandana</groupId>
 <artifactId>helloworld</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>

 <name>helloworld</name>
 <description>Demo project for Spring Boot</description>

 <parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>1.5.6.RELEASE</version>
  <relativePath/> <!-- lookup parent from repository -->
 </parent>

 <properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  <java.version>1.8</java.version>
  <spring.version>4.3.9.RELEASE</spring.version>
 </properties>

 <dependencies>
  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
  </dependency>

  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-test</artifactId>
   <scope>test</scope>
  </dependency>
 </dependencies>

 <build>
  <plugins>
   <plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
   </plugin>
  </plugins>
 </build>


</project>

HelloWorldController class:

package com.chandana.helloworld;

import com.chandana.helloworld.bean.Greeting;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/api")
public class HelloWorldController {

    @RequestMapping("/")
    public String welcome() {//Welcome page, non-rest
        return "Welcome to RestTemplate Example.";
    }

    @RequestMapping("/hello/{name}")
    public Greeting message(@PathVariable String name) {

        Greeting  msg = new Greeting(name, "Hello " + name);
        return msg;
    }

}

Conclusion: As it seems, it is very straightforward to implement RESTFul services using Spring Boot. So I got an idea to implement backend of my “Yield Price Sri Lanka” android app using Spring Boot. Besides, hoping to implement an Admin UI to manage price and commodity information and also a public web UI to display price details for users who don't have an Android app. Keep in touch.

Java 8 lambda expression for list/array conversion

2017-08-08T03:45:00.003+09:30

1). Convert List to List ( List of Strings to List of Integers)

List<Integer> integerList = stringList.stream().map(Integer::parseInt).collect(Collectors.toList());

// the longer full lambda version:

List<Integer> integerList = stringList.stream().map(s -> Integer.parseInt(s)).collect(Collectors.toList());

2). Convert List to int[](List of Strings to int array)

int[] intArray = stringList.stream().mapToInt(Integer::parseInt).toArray();

3). Convert String[] to List ( String array to List of Integers)

List<Integer> integerList = Stream.of(array).map(Integer::parseInt).collect(Collectors.toList());

4). Convert String[] to int[] (String array to int array)

int[] intArray = Stream.of(stringArray).mapToInt(Integer::parseInt).toArray();

5). Convert String[] to List (String array to Double List)

List<Double> doubleList = Stream.of(stringArray).map(Double::parseDouble).collect(Collectors.toList());

6). Convert int[] to String[] (int array to String array)

String[] stringArray = Arrays.stream(intArray).mapToObj(Integer::toString).toArray(String[]::new);

7). Convert 2D int[][] to List> ( 2D int array to nested Integer List)

List<Integer> list = Arrays.stream(dataSet).map(Arrays::asList).collect(Collectors.toList());

Manage Solr Data in WSO2 Server

2017-08-01T11:47:00.000+09:30

Recently I was checking an issue faced by one of my colleague while automating WSO2 API Manager deployment. There, once the new pack is deployed by pointing to the existing databases, APIM Store didn't show existing APIs at once. It took some time to display all the existing APIs in the Store.

The APIs are retrieved using the Solr based indexing in APIM. Therefore, the main reason for this behavior is that a fresh pack doesn't have existing Solr data and it takes some time to complete the indexing. Until that indexing process is completed, it will not show API in the Store instantly.

To address this, you can follow one of the below approaches:

1). Backup existing Solr data (APIM_HOME/solr/data) from the existing deployment and added it to newly created pack.

2). Externalize Solr data directory. Solr data stored location can be configured via core.properties file located in the APIM_HOME/repository/conf/solr/registry-indexing directory. So you can update core.properties to store Solr data outside the product directory.