Spring Boot Batch Parallel Jobs

Docker push denied requested access to the resource is denied unauthorized authentication required

2023-05-19T04:23:00.001-07:00

Open command prompt and execute below command

docker login

If you have Docker Desktop installed then it will automatically get credentials otherwise you can follow steps: click here

Ensure you have tagged the local images with the Docker account namespace -

To get Docker account namespace: click here

You can change the tag using below command
docker tag name-of-image docker_a/c_namespace/name-of-image

Example- C:\Windows\System32>docker tag eazybytes/loans ctcsinformation/loans

Finally you can push the docker images to docker hub

Right way to close HttpsURLConnetion using try with resources in java 8

2023-05-18T03:43:00.002-07:00

HttpsURLConnection does not implement the AutoCloseable interface in Java 8 or later versions.
Since HttpsURLConnection does not implement AutoCloseable, you cannot use it directly in a try-with-resources block. However, you can create a wrapper class that implements AutoCloseable and encapsulates the HttpsURLConnection object, and then use the wrapper class in a try-with-resources block.
Here's an example of how to create a wrapper class for HttpsURLConnection that implements AutoCloseable:

 
public class HttpsURLConnectionWrapper implements AutoCloseable {

    private final HttpsURLConnection connection;

    public HttpsURLConnectionWrapper(URL url) throws IOException {
        connection = (HttpsURLConnection) url.openConnection();
    }

    public HttpsURLConnection getConnection() {
        return connection;
    }

    @Override
    public void close() throws IOException {
        connection.disconnect();
    }
}

In this example, the HttpsURLConnectionWrapper class encapsulates the HttpsURLConnection object and implements the close() method to disconnect the connection. You can use this wrapper class in a try-with-resources block like this:

 
try (HttpsURLConnectionWrapper connectionWrapper = new HttpsURLConnectionWrapper(url)) {
    HttpsURLConnection connection = connectionWrapper.getConnection();
    // Use the connection here
} catch (IOException e) {
    // Handle the exception here
}

In this example, the HttpsURLConnectionWrapper object is created inside the try-with-resources block and automatically closed when the block exits. The getConnection() method returns the underlying HttpsURLConnection object, which can be used in the try block. When the block exits, the close() method is called on the wrapper object, which disconnects the connection.

How to create dynamic steps in Spring Batch

2021-04-20T07:22:00.001-07:00

We can create steps dynamically and return the job reference. Here is an example -


@Bean
public Job job() {
    Step[] stepsArray = // create your steps array or pass it as a parameter
    SimpleJobBuilder jobBuilder = jobBuilderFactory.get("mainCalculationJob")
            .incrementer(new RunIdIncrementer())
            .start(truncTableTaskletStep());
    for (Step step : stepsArray) {
        jobBuilder.next(step);
    }
    return jobBuilder.build();
}

Sending Bulk SMS using Sprint Boot Batch

2021-04-11T02:03:00.006-07:00

You must use Spring Boot Batch Jobs to send the bulk SMS's. It's very flexible to schedule these jobs using Cron expression, so you can send the SMS as per the requirement. e.g. daily, weekly or at a particular time.

We need to integrate our job with the SMS provider like - msg91

Following is the Job configuration - This class has reader which will read the data for SMS content and job defination


@Configuration
@EnableBatchProcessing
public class SmsBatchConfiguration {
	
	@Autowired
	@Qualifier("smsContentReader")
    
	public ItemReader> itemReader;
	
	@Autowired
	@Qualifier("nenewalNotificationSMSProcessor")
	RenewalNotificationSMSProcessor renewalNotificationSMSProcessor;
	
	@Autowired
	@Qualifier("smsSender")
	SmsSender SmsSender;
	
	@Autowired
	public JobBuilderFactory jobBuilderFactory;
	
	@Autowired
	public StepBuilderFactory stepBuilderFactory;
	
	@Value("${renewal.batch.notifications.email}")
	private String email;
	
	@Bean
    public JobExecutionListener listener1() {
        return new JobCompletionNotificationListener(email, "SMS_JOB");
    }
	
	@Bean("smsReminderJob")
    public Job renewalSMSNotificationJob() {
		return jobBuilderFactory.get("renewalSMSNotificationJob")
                .incrementer(new RunIdIncrementer())
                .start(getSmsRenewalPolicyData())
				.listener(listener1())
                .build();
    }

    @Bean
    public Step getSmsRenewalPolicyData() {
        return stepBuilderFactory.get("renewal-policy-sms-n")
                ., SmsFlowModel>chunk(1)
                .reader(itemReader)
                .processor(renewalNotificationSMSProcessor)
                .writer(SmsSender)
                .build();
    }
    
}

Following class is the item processor which will get the data from reader and pass it to writter


@StepScope
@Component("nenewalNotificationSMSProcessor")
public class RenewalNotificationSMSProcessor implements ItemProcessor, SmsFlowModel> {

    private static final Logger log = LoggerFactory.getLogger(RenewalNotificationSMSProcessor.class);

    @Value("${renewal.batch.notifications.sms.template_id}")
    private String template_id;

    @Value("${renewal.batch.notifications.sms.sender}")
    private String sender;

    @Override
    public SmsFlowModel process(List renewals) throws Exception {
        log.info("started - processing renewal policy details - SMS");
        List recipientsList = null;
        SmsFlowModel smsFlowModel = null;

        if (renewals != null && !renewals.isEmpty()) {
            recipientsList = new ArrayList<>();
            for (Renewal renewal : renewals) {
                if (renewal.getContactNumber() != null) {   // You will get the mobile number of recipient
                    MessageParameters msgParam = new MessageParameters();
                    msgParam.setPolicyno(renewal.getPolicyNo());
                    msgParam.setPremiumamount(String.valueOf(renewal.getPremium()));
                    msgParam.setMobiles("91".concat(renewal.getContactNumber().trim()));
                    recipientsList.add(msgParam);
                }
            }
            if (!recipientsList.isEmpty()) {
                smsFlowModel = new SmsFlowModel();
                smsFlowModel.setTemplate_id(this.template_id);
                smsFlowModel.setSender(this.sender);
                smsFlowModel.getRecipients().addAll(recipientsList);
            }
            return smsFlowModel;
        }
        return null;
    }

}

Following class will send the SMS, will are invoking the msg91 api to send the SMS to mobile numbers


@StepScope
@Component("smsSender")
public class SmsSender implements ItemWriter {
    private static final Logger LOGGER = LoggerFactory.getLogger(SmsSender.class);

    @Value("${renewal.batch.notifications.sms.auth}")
    private String authKey;

    @Value("${renewal.batch.notifications.sms.url}")
    private String apiUrl;

    @Override
    public void write(List bulkSMS) throws Exception {
        LOGGER.info("about to start sending SMS---");
        StringBuilder msg = (StringBuilder) CommonConstant.getDaysMap().get("sms-job-message");

        if (bulkSMS != null && bulkSMS.size() > 0) {
            SmsFlowModel sms = bulkSMS.get(0);
            ObjectMapper mapper = new ObjectMapper();
            String jsonString = mapper.writeValueAsString(sms);
            msg.append("\n").append("SMS request json : ").append(jsonString);

			HttpResponse response = null;

            try {
                response = Unirest.post(apiUrl)
                        .header("authkey", authKey)
                        .header("content-type", "application/json")
                        .body(jsonString)
                        .asJson();
            } catch (UnirestException e) {
                LOGGER.error("Error occured while sending sms : {}", e.getMessage());
                e.printStackTrace();
            }

			msg.append("\n").append("------------------------------------------------------------------------------------");
			if (response != null) {
                JSONObject myObj = response.getBody().getObject();
                LOGGER.info("SMS server response : " + myObj);
                msg.append("\n").append("SMS server response : ").append(myObj);
            } else {
                LOGGER.info("SMS server response response is null");
                msg.append("\n").append("SMS server response response is null");
            }

        } else {
            LOGGER.info("--sms sender object is null ---");
            msg.append("\n").append("sms sender object is null");
        }
        CommonConstant.getDaysMap().put("sms-job-message", msg);
    }
}

Properties file

This job will send the SMS on daily basis at 12 PM (as per the cron expression value)



##Batch properties##
spring.profiles.active=prod
spring.batch.job.enabled=false
spring.first.initialization=false

##Database##
spring.datasource.url=jdbc:mysql://localhost:3306/test_db
spring.datasource.username=uname
spring.datasource.password=passwd
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.initialization-mode=ALWAYS
spring.datasource.continue-on-error=false
spring.jpa.database-platform=org.hibernate.dialect.MySQL5InnoDBDialect

spring.jpa.properties.hibernate.format_sql=false
spring.jpa.hibernate.ddl-auto=none
spring.jpa.show-sql=false

#sms-server
renewal.batch.notifications.sms.auth=
renewal.batch.notifications.sms.url=http://api.msg91.com/api/v5/flow/?response=json
renewal.batch.notifications.sms.template_id=
renewal.batch.notifications.sms.sender=SHRI

#cron-expression
sms.job.launch.cronExpression=0 0 12 * * *

POM file link https://github.com/shrikantar/bulksms/blob/main/pom.xml

Git Commands -

2019-07-01T05:43:00.002-07:00

Commands	Description
`git status`	`Check status`
`git add [file-name.txt]`	`Add a file to the staging area`
`git add` `–A`	`Add all new and changed files to the staging area`
`git commit -m "[commit message]"`	`Commit changes`
`git rm -r [file-name.txt]`	`Remove a file (or folder)`
`git branch`	`List branches (the asterisk denotes the current branch)`
`git branch` `–a`	`List all branches (local and remote)`
`git branch [branch name]`	`Create a new branch`
`git branch -d [branch name]`	`Delete a branch`
`git push origin --delete [branch name]`	`Delete a remote branch`
`git checkout -b [branch name]`	`Create a new branch and switch to it`
`git checkout -b [branch name] origin/[branch name]`	`Clone a remote branch and switch to it`
`git checkout [branch name]`	`Switch to a branch`
`git checkout -`	`Switch to the branch last checked out`
`git checkout -- [file-name.txt]`	`Discard changes to a file`
`git merge [branch name]`	`Merge a branch into the active branch`
`git merge [source branch] [target branch]`	`Merge a branch into a target branch`
`git stash`	`Stash changes in a dirty working directory`
`git stash clear`	`Remove all stashed entries`
`git push origin [branch name]`	`Push a branch to your remote repository`
`git push -u origin [branch name]`	`Push changes to remote repository (and remember the branch)`
`git push`	`Push changes to remote repository (remembered branch)`
`git push origin --delete [branch name]`	`Delete a remote branch`
`git pull`	`Update local repository to the newest commit`
`git pull origin [branch name]`	`Pull changes from remote repository`
`git log`	`View changes`
`git log --summary`	`View changes (detailed)`
`git diff [source branch] [target branch]`	`Preview changes before merging`

Git Quick Card

2019-07-01T05:42:00.001-07:00

Git – Introduction

There are many popular DVCS systems like Git, Mercurial, Bazaar which has grown maturely to implement distributed revision across their products. Git is widely used by the development teams across the world due to its simplicity, active open source the development team and frequent releases/patches. Before moving forward with the git concepts, few of them terminologies which will be frequently used in this tutorial are defined below :

Remote Repository : This refers to the specific project space on the central server where the project source code is stored and will be used by the team members for checkout or commit.

Local Repository (.git directory): There is a local folder “.git” which is automatically created when the project or folder is git tracked. The “.git” folder is like a local database which will contain all information related to branches, commit history, user configurations, and metadata.

Branches / Tags – Has the same definition as that of SVN /VSS.

Checkout – Since git has a local copy of all branches, a checkout of a branch would always be a local operation and not a remote server call as in SVN.

Clone – When a git clone command is invoked, the entire remote repository is fetched which includes the branches, tags, commit history and file changes.

Staging Area - is a file located in the project git directory, that stores information about what will go into your next commit.

Spring batch introduction

2019-02-12T10:58:00.000-08:00

An open source framework to create batch processing application which can deal with the high volume of data, capable to handle processing of billions of records in a day efficiently.
Lightweight framework – consume very less memory.
It’s not a scheduling framework
Provide out of the box APIs to read, process and write data.
Efficient transaction management and error handling.

Why does the batch job needed?

Automate the complex processing (applying business logic on data/transforming data) of large volumes of data without any user interaction.

Spring Batch vs Spring Boot Batch -

·Spring batch is a batch processing framework and spring boot help us to do rapid development by using starter-templates (convention over configuration).

What are the ideal use cases for the batch job?

Reads the high volume of records from the database/file system.
Process the data e.g. Month-end / Weekend calculations
Write back processed data / generates reports

What are the different ways to achieve high-performance batch processing?

Parallel steps
Multithreaded steps
Partitioning

What are the advantages of spring batch?

· Easy to configure, customize by leveraging the spring framework in all layers.

· Easy to build and deploy.

· Developers only needs to focus on business logic other things will be taken care by the framework, like records processed, how many records are remaining to process how to commit those records etc.

Spring Boot Batch - Parallel Processing (Running Multiple Jobs Concurrently)

2019-02-09T12:19:00.001-08:00

If you are new to Spring batch please check the earlier post about Spring batch introduction

In this post we will see Spring boot batch example, Spring boot batch parallel processing example

Spring boot batch parallel processing allows multiple batch jobs to run in parallel to minimize the total elapsed batch processing time.

This is not a problem as long as the jobs are not sharing the same files, db-tables, or index spaces. If they do, this service should be implemented using partitioned data.

We will create the application having the following 2 jobs -

1st Job: Reads the Product information from the database and after processing will give the rating to the project.

2nd Job: Reads the Invoice the information the database and update back to the database.

We are not focusing on the business logic here but the processing of data using multiple jobs.

Technical Details -

Java 8
Spring Boot 2.x
Spring Batch 4.x
MySql 5.7.17
IntelliJ-Community Edition 2018-3 (Same code will work in Eclipse as well 😃

Source Code available on GitHub -

https://github.com/shrikantar/spring-boot-batch-parallel-jobs.git

Project Structure -

InvoiceProcessingJob and ProductProcessingJob these are 2 jobs having their own ItemReader, ItemProcessor, and ItemWriter.
JobLauncherConfig will run both the above job

pom.xml Following are the dependencies -

application.propeties -

By default Spring batch is enabled, it means that as soon as application started, job will launch

We are invoking our job using scheduler that why we have provided spring.batch.job.enabled=false

Job-1: We have created the job. Notice the various annotations we are using here. @Configuration required to inform Spring container this is a Config file and will hold bean definitions.

@EnableBatchProcessing - Enables the batch processing.

Chunk Size - Denotes the commit interval. After processing 5 records job will commit the transaction

We are using the JdbcCursorItemReader which will read the data from the database -

Job-2: This job reads the product information from the database and generates the invoices

We will use the following class to launch the job we have created above. We have autowired both jobs and using job launcher will start them. The launch method will get periodically called as we have annotated with the @Scheduled annotation

Finaly Job Launcher -

Finally following is the main class to start our Spring Boot batch application -