Learn Java Skills

Mockito Exception Throwing for Unit Testing | Learn Java Skills

2025-02-02T19:42:00.000+05:30

There is a standard coding practice in Java application development to capture the exception using the try-catch block, and there may be some business logic in the catch block that you wish to unit test.

If you are wondering, is there a possibility of using the mockito framework to test the catch block?

Mockito makes it simple to throw an exception when invoking the method.

In this quick lesson, we'll look at how to throw an exception and unit test the catch block. Alternatively, you may test your business logic to examine how exceptions are handled in your application.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Code Overview

In this lesson, we'll look at how we may throw exceptions on both the void and non-void methods.

As a result, I created the following code to cover both functions: I'll teach you how to throw an exception on the non-void notificationService.sendNotification() method and the void transactionRepository.saveNetBankingTransaction() method.

 package in .learnjavaskills.mockitotutorial.service;

 import in .learnjavaskills.mockitotutorial.repository.TransactionRepository;

 import java.math.BigDecimal;
 import java.util.Objects;
 import java.util.concurrent.TimeUnit;

 public class TransactionService {
    private TransactionRepository transactionRepository;

    private NotificationService notificationService;
    public TransactionService(TransactionRepository transactionRepository, NotificationService notificationService) {
        this.transactionRepository = transactionRepository;
        this.notificationService = notificationService;
    }

    public void netBankingTransaction(String username, String password,
        BigDecimal amount, Long transferAccountNumber) throws InterruptedException {
        boolean isValidUser = authenticateUser(username, password);
        if (isValidUser) {
            boolean isTransferSuccess = transferAmount(transferAccountNumber, username, amount);
            if (isTransferSuccess) {
                // Send notification to user
                try {
                    boolean isNotificationSend = notificationService.sendNotification("toEmail@address.com", "Transaction success for " + amount, "Hi " + username + " \n Successfully transfer amount " + amount);
                } catch (Exception exception) {
                    TimeUnit.SECONDS.sleep(10L);
                    // Re-Trying sending notification
                    boolean isNotificationSend = notificationService.sendNotification("toEmail@address.com", "Transaction success for " + amount, "Hi " + username + " \n Successfully transfer amount " + amount);
                }

                // save net banking transaction
                try {
                    transactionRepository.saveNetBankingTransaction(username, amount);
                } catch (Exception exception) {
                    TimeUnit.SECONDS.sleep(10L);
                    // Re-Trying to save net banking transaction
                    transactionRepository.saveNetBankingTransaction(username, amount);
                }
            }
        }
    }
 }

Non-Void Return Type Method

Let's start by learning how to throw an exception on the non-void return type method. This is as simple as mocking the object and returning some dummy value.

In the following example, I've mock the NotificationService class's sendNotification() method, which is in charge of notifying the user. According to the business logic in the above code, if we find an error while notifying the end user, it will retry notifying them after 10 seconds.

Let's see how we can mock the sendNotification() function to throw an exception and check that the sendNotification() method should be executed twice according to our business logic.

 package in .learnjavaskills.mockitotutorial.service;

 import in .learnjavaskills.mockitotutorial.exception.TransactionException;
 import in .learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest {
    @Mock
    private TransactionRepository transactionRepository;

    @Mock private NotificationService notificationService;

    @InjectMocks
    private TransactionService transactionService;

    @Test
    void testWhenExceptionThrownWileSendingNotification() throws InterruptedException {
        BDDMockito.given(notificationService.sendNotification(ArgumentMatchers.any(), ArgumentMatchers.any(),
                ArgumentMatchers.any()))
            .willThrow(TransactionException.class);

        // verifying excetion is thrown with TransactionException
        Assertions.assertThatThrownBy(() - > transactionService.netBankingTransaction("LearnJavaSkills.in",
                "password", BigDecimal.TEN, 1234567890L))
            .isExactlyInstanceOf(TransactionException.class);

        //  verifying 2 times sendNotification() method should invoke because in catch block also we are re-trying to send notification
        BDDMockito.verify(notificationService, Mockito.times(2))
            .sendNotification(ArgumentMatchers.any(), ArgumentMatchers.any(), ArgumentMatchers.any());
    }
 }

Void Return Type Method

Let's look at how we may throw an exception on the void method now. Throwing an exception on the void return type method differs somewhat from the non-void return type method shown in the previous example.

To test the catch block code, we mock transactionRepository to raise a TransactionException when the saveNetBankingTransaction() method is executed.

If you thoroughly examined the code overview, you could have seen that if any errors are encountered during saving the net banking transaction, it must re-try after 10 seconds to save the same transaction.

I also used the BDDMockito.verify() method to validate this situation.

 @Test
 void testWhenExceptionThrownWhileSavingNetBankingTransaction() {
    // mocking transactionRepository to throw TransactionException when saveNetBankingTransaction() invoke
    BDDMockito.doThrow(TransactionException.class)
        .when(transactionRepository)
        .saveNetBankingTransaction(ArgumentMatchers.any(), ArgumentMatchers.any());

    // verifying exception are thrown which mock in the above
    Assertions.assertThatThrownBy(() - > transactionService.netBankingTransaction("LearnJavaSkills.in",
            "password",
            BigDecimal.TEN,
            1234567890L))
        .isExactlyInstanceOf(TransactionException.class);

    // verifying saveNetBankingTransaction method invoke two times as per the business logic
    BDDMockito.verify(transactionRepository, Mockito.times(2))
        .saveNetBankingTransaction(ArgumentMatchers.any(), ArgumentMatchers.any());
 }

How to Throw an Exception with a Message

Mockito allows us to throw exceptions as objects, therefore we can also throw exceptions with messages.

In our earlier example, we may substitute the Exception object for the Exception class.

Non-Void Return Type Method - Throw an Exception with a Message

 @Test
 void testWhenExceptionThrownWileSendingNotification() throws InterruptedException {
    String message = "Unable to send notification to user";
    BDDMockito.given(notificationService.sendNotification(ArgumentMatchers.any(), ArgumentMatchers.any(),
            ArgumentMatchers.any()))
        .willThrow(new TransactionException(message));

    Assertions.assertThatThrownBy(() - > transactionService.netBankingTransaction("LearnJavaSkills.in",
            "password",
            BigDecimal.TEN,
            1234567890L))
        .isExactlyInstanceOf(TransactionException.class)
        .hasMessageContaining(message);

    BDDMockito.verify(notificationService, Mockito.times(2))
        .sendNotification(ArgumentMatchers.any(), ArgumentMatchers.any(), ArgumentMatchers.any());
 }

Void Return Type Method - Throw an Exception with a Message

  @Test
 void testWhenExceptionThrownWhileSavingNetBankingTransaction() {
     // mocking transactionRepository to throw TransactionException when saveNetBankingTransaction() invoke
     String message = "unable to save net banking transaction";
     BDDMockito.doThrow(new TransactionException(message))
         .when(transactionRepository)
         .saveNetBankingTransaction(ArgumentMatchers.any(), ArgumentMatchers.any());

     // verifying exception are thrown which mock in the above
     Assertions.assertThatThrownBy(() - > transactionService.netBankingTransaction("LearnJavaSkills.in",
             "password",
             BigDecimal.TEN,
             1234567890L))
         .isExactlyInstanceOf(TransactionException.class)
         .hasMessageContaining(message);

     // verifying saveNetBankingTransaction method invoke two times as per the business logic
     BDDMockito.verify(transactionRepository, Mockito.times(2))
         .saveNetBankingTransaction(ArgumentMatchers.any(), ArgumentMatchers.any());
 }

Stubbing Multiple Calls

Mockito is one of the most powerful Java mocking frameworks. Mockito allows us to stub numerous times with various behaviors, which is highly useful for testing sequential exception handling or forward-thinking behavior.

If you've been paying attention to our code overview, you'll see that we're retrying to send the notification to the enduser if any errors occur at notifying.

With mockito, you may stub this sendNotification() method to throw an exception, and then stub this sendNotification() method again to return values when sendNotification() is executed the second time.

Let's look at how we'll unit test if we're willing to throw an exception on the initial method called and return values on the re-try attempt.

 @Test
 void testWhenExceptionThrownWileSendingNotificationStubMultiCall() throws InterruptedException {
    String message = "Unable to send notification to user";
    BDDMockito.given(notificationService.sendNotification(ArgumentMatchers.any(), ArgumentMatchers.any(),
            ArgumentMatchers.any()))
        .willThrow(new TransactionException(message))
        .willReturn(true);

    transactionService.netBankingTransaction("LearnJavaSkills.in",
        "password",
        BigDecimal.TEN,
        1234567890L);

    BDDMockito.verify(notificationService, Mockito.times(2))
        .sendNotification(ArgumentMatchers.any(), ArgumentMatchers.any(), ArgumentMatchers.any());
 }

Conclussion

Mockito is a powerful mocking framework for Java applications. We can use the mockito framework to not only mock objects such that they return the values we want, but we can also throw an exception on method call by mocking an object.

Mockito allows multiple stubbing calls, which implies that if you have numerous calls to that method, you may control the behavior by stubbing all of them at once.(alert-success)

Multiple stubbing works in combination with the sequential method invocation in your business logic. Assume you've called a method three times, the first two times throwing an exception and the third time being requested to return values.

The method will then throw an exception the first two times it is executed in your business logic, and it will return values the third time.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

(getButton) #text=(Next: Mock constructor in Mockito) #icon=(link) #color=(#2339bd)

How to mock constructor in Mockito

2025-02-02T19:38:00.001+05:30

Creating new objects directly within a method often violates dependency injection principles. Ideally, refactoring to adhere to clean code architecture is the preferred solution.

However, if refactoring isn't immediately feasible, the following example demonstrates how to mock object creation (constructors) in Java using Mockito.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

TransactionService class

The upiPayment() method within my TransactionService class relies on the Account class to check if UPI payments are enabled for a given account number.

To avoid invoking the real isUpiAllowed() method during testing, I intend to mock the Account constructor.

This will allow me to inject a mock Account object that provides a controlled response when its isUpiAllowed() method is called.

This mocking strategy is generally applicable and can be used with other dependencies, such as RestTemplate for mocking API responses.

 public class TransactionService
 {
   public boolean upiPayment(Long accountNumber, BigDecimal amount) {
     Account account = new Account(); // we'll mock this constructor
     boolean upiAllowed = account.isUpiAllowed(accountNumber, amount);
     if (upiAllowed) {
       System.out.println("transaction completed");
       return true;
     }
     return false;
   }
 }

 public class Account {
    public boolean isUpiAllowed(Long accountNumber, BigDecimal amount) {
      System.out.println("Checking is UPI transaction active for account " + accountNumber + " for amount " + amount);
      return Objects.nonNull(accountNumber) && accountNumber > 1
              Objects.nonNull(amount) && amount.compareTo(BigDecimal.valueOf(5000d)) < 0;
   }
 }

Mocking the Account Constructor for JUnit Tests

 @Test
 void upiTransaction() {
   try (MockedConstruction<Account> accountMockedConstruction = Mockito.mockConstruction(Account.class, (mock, context) ->
          Mockito.when(mock.isUpiAllowed(ArgumentMatchers.anyLong(), ArgumentMatchers.any()))
                      .thenReturn(true))) {
     boolean transactionStatus = transactionService.upiPayment(1234567890L, BigDecimal.TEN);
     assertTrue(transactionStatus);
   }
 }

Mockito.mockConstruction() method: Explanation

Mockito's mockConstruction() method provides a powerful way to mock the instantiation of new objects, offering greater flexibility than older approaches like whenNew().

It's particularly useful when dealing with classes you don't directly control, or when you need to mock the creation of objects within the system under test.

This approach allows you to intercept and control the behavior of newly created instances, enabling you to isolate dependencies, simulate specific scenarios, and thoroughly test your code

The following overloaded version of the mockConstructor method is used in this example.

 public static <T> org.mockito.MockedConstruction<T> mockConstruction(java.lang.Class<T> classToMock, org.mockito.MockedConstruction.MockInitializer<T> mockInitializer) { /* compiled code */ }

MockInitializer interface: Explanation

MockInitializer in Mockito is a powerful mechanism that allows you to customize and extend the initialization process of your mocks.

It provides a way to execute specific setup logic before mocks are fully initialized, giving you fine-grained control over their behavior and state.

This is particularly useful for handling complex initialization scenarios, setting up default values, or integrating with other testing frameworks.

To customize our mock, we simply implement the following prepare() method.

 @java.lang.FunctionalInterface
 static interface MockInitializer <T> {
   void prepare(T t, org.mockito.MockedConstruction.Context context) throws java.lang.Throwable;
 }

 static interface Context {
   int getCount();

   java.lang.reflect.Constructor<?> constructor();

   java.util.List<?> arguments();
 }

Conclussion

In this tutorial, we've explored the powerful capabilities of Mockito for mocking constructors, a crucial technique for effective unit testing, especially when dealing with complex dependencies or legacy code.

We've seen how mockConstruction() provides a flexible and modern approach to intercept and control object creation, enabling us to isolate the system under test and simulate various scenarios.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

How to Mock Static Methods with Mockito: A Complete Guide | Learn Java Skills

2025-01-26T20:48:00.000+05:30

You might need to mock the static method in your unit test in some circumstances. In this brief tutorial, we'll learn how to utilize the mockito framework to mock the behavior of a Java static method. Now, version 3.4.0 or later supports mocking abalibity.

When designing clean, maintainable, and object-oriented code, it's usual practice to avoid mock static methods because doing so could interfere with the project architecture.

One of the foundational principles of unit testing is that in order to use it, code must be written in a specific way that makes it testable.

This can be accomplished by using a concept known as dependency injection, or by specifying dependencies as interfaces that can be used in place of "fake implementations" during testing.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Static Utility Class

In order to mock the static method, I developed a simple utility class with two static methods, getTodayDay() and getDayFromDate(LocalDate localDate).

Both methods are used to determine the day of the week. I've written two methods to demonstrate how to mock the static method without an argument and the static method with an argument.

Utility class

 package in.learnjavaskills.mockitotutorial.utils;

 import in.learnjavaskills.mockitotutorial.enums.WeekDays;

 import java.time.LocalDate;
 import java.util.Calendar;

 public class Utility
 {
    public static WeekDays getTodayDay()
    {
        int day = Calendar.getInstance().get(Calendar.DAY_OF_WEEK);
        return WeekDays.getWeekDay(day);
    }

    public static WeekDays getDayFromDate(LocalDate localDate)
    {
        int day = localDate.getDayOfWeek().getValue();
        return switch (day) {
            case 1, 2, 3, 4, 5, 6 -> WeekDays.getWeekDay(day+1);
            case 7 -> WeekDays.SUNDAY;
            default -> WeekDays.UNKNOWN;
        };
    }
 }

WeekDays enum

 package in.learnjavaskills.mockitotutorial.enums;

 public enum WeekDays
 {
    SUNDAY(1), MONDAY(2), TUESDAY(3), WEDNESDAY(4), THURSDAY(5), 
    FRIDAY(6), SATURDAY(7), UNKNOWN(0);

    private int day;

    WeekDays(int day)
    {
        this.day = day;
    }

    public static WeekDays getWeekDay(int day)
    {
        for (WeekDays weekDays : WeekDays.values())
            if (weekDays.day == day)
                return weekDays;
        return WeekDays.UNKNOWN;
    }
 }

MockedStatic<T> Interface

The MockedStatic<T> interface, which is included in the org.mockito package, represents an active mock of a type's static methods.

The mocking only impacts the thread on which this static mock was produced, therefore using this object from another thread is not safe.

It is advised that you use the ScopedMock.close() method to release the static mock. If the static mock is never closed, it will continue active on the current process.

Since MockedStaticc<T> extends the ScopedMock interface farther than the AutoCloseable interface, it is advised to utilize it within the try-with resource, which will be auto-closed.

Mocking the No-Argument static method

Mock the first static method, getTodayDay(), which returns an enum of WeekDays.

To test the static mocking behavior of the MockedStatic<T> Interface, I'll use the getTodayDay() method without mocking to retrieve an actual day of the week, and then mock it inside the try-with resource scope to return the other day.

Once the scope of try-with resources has been fulfilled, we will call the getTodayDay() method again to see if the mock has been released.

 package in.learnjavaskills.mockitotutorial.utils;

 import in.learnjavaskills.mockitotutorial.enums.WeekDays;
 import org.junit.jupiter.api.Test;
 import org.mockito.BDDMockito;
 import org.mockito.MockedStatic;
 import org.mockito.Mockito;

 import java.time.LocalDate;

 import static org.junit.jupiter.api.Assertions.*;

 class UtilityTest
 {
    
    @Test
    void mockGetTodayDay()
    {
        // testing actual day without mocking
        assertEquals(WeekDays.TUESDAY, Utility.getTodayDay());

        try (MockedStatic<Utility> utilityMockStatic = Mockito.mockStatic(Utility.class))
        {
            // mocking the getTodayDay method to get response of the day of week as sunday instead of today day
            utilityMockStatic.when(Utility :: getTodayDay)
                    .thenReturn(WeekDays.SUNDAY);

            assertEquals(WeekDays.SUNDAY, Utility.getTodayDay());

            // verify getTodayDay method invoke at least one time
            utilityMockStatic.verify(Utility :: getTodayDay);
        }

        // testing is mock release after the try-with resource scope
        assertEquals(WeekDays.TUESDAY, Utility.getTodayDay());
    }
 }

Mocking the Static Method with Arguments

Mocking the static method with the argument is identical to mocking the static method without the argument.

Let's first grasp the Verification Interface before mocking the static method with the parameter.

Verification Interface

The Verification interface is a functional interface with a single abstract method, void apply(), which throws Throwable. It's in the org.mockito package and is used to validate the static method.

The MockedStatic<T> interface's when() method required the Verification in the parameter and syntax given below.

 <S> OngoingStubbing<S> when(Verification verification);

We utilized the method reference in the previous example of mocking the static method, but if you wish to mock the argument static method, you must implement the apply() abstract method of the Verification interface using the lambda expression.

Program on Mocking the Static Method with Arguments

 package in.learnjavaskills.mockitotutorial.utils;

 import in.learnjavaskills.mockitotutorial.enums.WeekDays;
 import org.junit.jupiter.api.Test;
 import org.mockito.ArgumentMatchers;
 import org.mockito.BDDMockito;
 import org.mockito.MockedStatic;
 import org.mockito.Mockito;

 import java.time.LocalDate;

 import static org.junit.jupiter.api.Assertions.*;

 class UtilityTest
 {
    @Test
    void mockGetDayFromDate()
    {
        LocalDate localDate = LocalDate.now();
        // testing getDayFromDate without mocking
        assertEquals(WeekDays.TUESDAY, Utility.getDayFromDate(localDate));

        // let's mock the Utility.getDayFromDate to return Sunday.
        try(MockedStatic<Utility> utilityMockStatic = Mockito.mockStatic(Utility.class))
        {
            utilityMockStatic.when(()-> Utility.getDayFromDate(ArgumentMatchers.any()))
                    .thenReturn(WeekDays.SUNDAY);

            assertEquals(WeekDays.SUNDAY, Utility.getDayFromDate(localDate));

            // verify mock has been called
            utilityMockStatic.verify( ()-> Utility.getDayFromDate(localDate));
        }

        // check if mock has been release by invoking getDayFromDate outside try-with scope
        assertEquals(WeekDays.TUESDAY, Utility.getDayFromDate(localDate));
    }
 }

Conclussion

We learnt how to mock both sorts of static methods in this brief tutorial: static methods with parameters and static methods without arguments.

We recognize the MockedStatic<T> interface, as well as the parameter of the MockedStatic<T>'s when() method, i.e., the Verification Interface.

Always use MockedStatic<T> with the try-with resource, as it is critical to release the mocking by executing the ScopedMock interface's close() function.(alert-success)

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

(getButton) #text=(Next: Mockito Exception Throwing for Unit Testing) #icon=(link) #color=(#2339bd)

How to Mock Void Methods with Mockito: Everything You Need to Know | Learn Java Skills

2025-01-26T20:46:00.000+05:30

If you're a programmer, I'm sure you've written about the void method in your application, and as a best practice for writing unit tests of your application using the concepts of test-derived development and behavior-driven development, you might be wondering how we mock the void method.

If so, you've come to the right place. In this brief article, we will learn how to mock the void method in a Java program using mockito, the most popular unit testing framework.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Why is it necessary to mock the void method?

Let's start with why we need to mock the void method. To make your test case run faster, you should avoid invoking the void function in your unit testing. To accomplish this purpose, we have a doNothing() method.

The mockito's default behavior is to do nothing. You read that accurately. You might be questioning why we need doNothing() in the first place.

The rationale for this is so you may use the ArgumentCaptor to capture the void method's arguments and check that the correct values were supplied to the void method. You may accomplish this by combining doNoting() and the ArgumentCaptor.

You can also throw an exception at the time of invoking the void method to identify the behavior of your program by using the doThrow() method of the mockito.

There are numerous other actions that we can perform by mocking the void method, as we will see in the articles.

How to use the doNothing() method to mock the void method

I'm assuming you have the mockito framework properly configured in your Java application; if not, see what is Mockito post.

doNothing() is the static method of the Mockito class, which is extended by the BDDMockito class.

You may use doNothing() to instruct the void method to do nothing when invoked. The default behavior is to do nothing.

Here's an example of using doNothing with the mockito to mock the void method . In the following example, we limit the DB call by mocking the void method of the saveTransaction() method, which is responsible for storing the transaction data into the DB.

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

	@InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

	@Test
    void doNothingWhenSaveTransactionIsInvoked()
    {
        BDDMockito.doNothing()
                .when(transactionRepository)
                .saveTransaction(transactionDetail);

        transactionService.creditCardTransaction(transactionDetail);
    }
 }

How to use the doNothing() method with ArgumentCaptor to capture the arguments of the void method

In our previous example, we saw how we can use the doNothing() method to mock the void method onto doing nothing, and we also learned that doNothing is the default behavior.

As previously explained, we may capture the argument using the ArgumentCaptor by utilizing the doNothing() method.

The mockito class's doNothing() method is used to capture and verify the parameter in the example below.

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

    @Test
    void argumentCaptorWithDoNothingOnSaveTransaction()
    {
        ArgumentCaptor<TransactionDetail> transactionDetailArgumentCaptor = 
        	ArgumentCaptor.forClass(TransactionDetail.class);

        BDDMockito.doNothing()
                .when(transactionRepository)
                .saveTransaction(transactionDetailArgumentCaptor.capture());

        transactionService.creditCardTransaction(transactionDetail);

        TransactionDetail transactionDetailArgumentCaptorValue = 
        	transactionDetailArgumentCaptor.getValue();
        Assertions.assertThat(transactionDetailArgumentCaptorValue)
        	.isNotNull();
        Assertions.assertThat(transactionDetailArgumentCaptorValue.transactionAmount())
        	.isEqualTo(transactionDetail.transactionAmount());
        Assertions.assertThat(transactionDetailArgumentCaptorValue.transactionId())
        	.isEqualTo(transactionId);
    }
 }

How to use the doThrow() method to throw an exception on the void method

It's best practise to test your code to check how it reacts when exceptions are thrown.

To test how your code behaves, we may utilize the Mockito framework's doThrow() function to throw an exception when methods are invoked.

The doThrow() method of the Mockito class, which is extended by the BDDMockito class, is a static method. When you throw an exception using the doThrow() function, a new exception instance is produced for each method call.

The following is a basic example of utilizing the doThrow() method to throw an exception while using the saveTransaction() method to persist data into the database.

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

    @Test
    void doThrowWhenSaveTransactionInvoke()
    {
        BDDMockito.doThrow(ClassCastException.class)
                .when(transactionRepository)
                .saveTransaction(transactionDetail);

        Assertions.assertThatThrownBy(()-> transactionService.creditCardTransaction(transactionDetail))
                .isExactlyInstanceOf(ClassCastException.class);
    }
 }

How to use the doCallRealMethod() method to invoke the real implementation of the method

We already know that the Mockito framework's default approach when mocking the void method is to do nothing, but what if we actually want to execute the real implementation to test our code?

In that instance, we may use the doCallRealMethod() function to call the real method.

The following is a simple program for invoking the void method to test the implementation using the doCallRealMethod() function.

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

    @Test
    void doCallRealMethodWhenSaveTransactionInvoke()
    {
        BDDMockito.doCallRealMethod()
                .when(transactionRepository)
                .saveTransaction(transactionDetail);

        transactionService.creditCardTransaction(transactionDetail);

        BDDMockito.verify(transactionRepository, Mockito.times(1))
                .saveTransaction(transactionDetail);

    }
 }

Conclussion

When it comes to mocking the void method, the mockito framework's default approach is to do nothing. In such scenario, utilize the ArgumentCaptor with doNothing() to capture the void method's parameters and check whether or not the right values were supplied to the void method.

There may be times when you need to test if a void method throws an exception. In this case, doThow() enables us to raise an exception on the void method.

To test the void method's real implementation, we may completely rely on the doCallRealMethod() method, which can partially mock the void method and enable the the method's real implementation.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

(getButton) #text=(Next: How to Mock Static Methods with Mockito) #icon=(link) #color=(#2339bd)

What is Mockito? Everything You Need to Know About This Popular Java Mocking Framework | Learn Java Skills

2025-01-26T20:43:00.000+05:30

Mockito is a Java framework that allows you to mock or clone real-world objects.

In a nutshell, mockito is a BDD, or behavior-driven development, unit testing approach that allows you to verify the functionality of the code by mocking the objects rather than utilizing real objects.

With the mockito framework, we can not only mock or clone the object, but also control the behavior of the function.

Such as passing dummy values as input or instructing that if a specific method has been called on the given mocked object, simply return some dummy values instead of calling that function.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~(toc)~~

Why Mockito?

Mockito is one of the most popular mocking frameworks for Java, and it can be easily in combination with other popular unit testing frameworks in Java, such as JUnit and TestNG.

Mockito, a behavior-driven programming framework, has both an annotation base and a code base for mocking.

Mockito provides a wide range of functionalities, including stubbing method calls, verifying method calls, and validating method call order.

Mockito is capable of mocking complicated dependencies like as databases and web services.

Mockito allows you to test error handling using stub method calls that return errors. This enables you to test how the code in test handles errors. This can assist you in writing more robust code.

Dependencies are required to work with the mockito framework

Thanks to build tools including Gradle, Maven, etc., enabling behavior-driven development using the Mockito framework is now simple.

Simply include the script below in the build.gradle In order to have the mockito in your application work with Gradle.

testImplementation 'org.mockito:mockito-core:5.6.0'

The code below can be used to configure the mockito if you are using the Maven build tool:

 <dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-core</artifactId>
    <version>5.6.0</version>
    <scope>test</scope>
 </dependency>

If you are not familiar with Gradle or Maven, you may use any build tool, and mocktio library scripts can be found in the Maven central repository.

As previously explained, the mockito is simple to set up with other Java unit testing frameworks like as JUnit or TestNG.

Because I'm going to use JUnit 5 with mockito, I'll need to add one additional dependency to my project, which is stated below.

testImplementation 'org.mockito:mockito-junit-jupiter:5.6.0'

 <dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-junit-jupiter</artifactId>
    <version>5.6.0</version>
    <scope>test</scope>
 </dependency>

The Mockito JUnit Jupiter library script can be found in the Maven central repository.

Mockito annotations

There are several annotations used in the mockito, such as @Mock, @Spy, @Captor, @InjectMocksand @ExtendWith(MockitoExtension.class), which we will discuss subsequently.

@Mock annotation

The @Mock annotation is used to mock or clone the object, which means you may change its behavior, such as function input, or supply some dummy output when it calls particular methods using the BDDMockito's when() and then() method.

The @Mock annotation is included in the org.mockito.Mock package.

@Mock private EmployeRepository employeeRepository;

@Captor annotation

The @Captor annotation can be used to capture the arguments' values. The @Captor annotations are used with the BDDMocktio class's verify() method.

The @Captor annotation is commonly used to validate that the same values have been supplied to the method you anticipate when unit testing by simply capturing the value using the BDDMOckito's verify method.

The org.mockito.Captor package contains the @Captor annotation.

@Captor private ArgumentCaptor<String> name;

@Spy annotation

The @Spy annotation may generate a partial fake object, which means that if you're spying on a list object, you can easily add the element to the real object (rather than the mock), and you can control the behavior of the output by stubging it using the when() and then() methods of the BDDMockito class.

The org.mockito.Spy package contains the @Spy annotation.

@Spy private ArrayList<String> listOfEmployeeName;

@InjectMocks annotation

The dependencies are injected into the object using the @InjectMocks annotation. The objects associated with the annotations @Mock and @Spy will be injected through the @InjectMocks annotation.

Dependency injection in the mockito framework can be achieved by the use of setters, constructor bases, or field injections, often known as property injections.

The @InjectMocks is available in the org.mockito.InjectMocks package and the following snippet show how to use the @InjectMocks annotation.

@InjectMocks private EmployeeService employeeService;

@ExtendWith(MockitoExtension.class) annotation

@ExtendWith annotation is a class-level JUnit 5 annotation that describes an extension model.. The MockitoExtension.class is used alongside with the @ExtendWith annotation to interact with JUnit5 where mock objects are initialized and strick stubbing is handled.

The @ExtendWith annotation is comparable to the MockitoJUnitRunner annotation in JUnit 4.

@ExtendWith is available in the org.junit.jupiter.api.extension.ExtendWith package, and MockitoExtension.class is available in the org.mockito.junit.jupiter.MockitoExtension package.

The code below demonstrates how to use @ExtendWith(MockitoExtension.class).

 @ExtendWith(MockitoExtension.class)
 class EmployeeServiceTest
 {

 }

Understand the classes that will be utilized for creating BDD test cases in this lecture.

We have a TransactionService class with the creditCardTransaction method, and we are going to generate test cases using the Mockito framework and behavior-driven development.

Before we go, let's take a short look at the classes utilized in this lesson.

TransactionService class

We are interested about developing BDD (behavior-driven development) unit test cases for the TransactionService class using the mockito framework.

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.exception.TransactionException;
 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.dto.TransactionStatus;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;

 import java.util.Objects;

 public class TransactionService
 {
    private TransactionRepository transactionRepository;
    public TransactionService(TransactionRepository transactionRepository)
    {
        this.transactionRepository = transactionRepository;
    }

    public TransactionStatus creditCardTransaction(TransactionDetail transactionDetail)
            throws TransactionException
    {
        if (Objects.isNull(transactionDetail))
            throw new TransactionException("transactionDetails must be non null");

        boolean uniqueTransactionId = transactionRepository.isUniqueTransactionId(transactionDetail.transactionId());
        System.out.println("is uniqueTransactionId : " + uniqueTransactionId);
        if (!uniqueTransactionId)
            throw new TransactionException("transaction id must be unique");

        boolean isTransactionSuccess = completeTransaction(transactionDetail);
        if (isTransactionSuccess)
            return new TransactionStatus((short) 200, "success");
        return new TransactionStatus((short) 501, "fail");
    }

    private boolean completeTransaction(TransactionDetail transactionDetail)
    {
        if (transactionDetail.cardNumber() < 0)
            return false;

        transactionRepository.saveTransaction(transactionDetail);
        return true;
    }
 }

TransactionRepository class

I designed a sample repository, TransactionRepository, with several hardcoded tasks. Typically, we develop the dao layer, which is in charge of interfacing with our database to fetch or save data as needed.

I wanted to show you how we limit the DB call while designing the unit test case since DB calls in the unit test case might be quite expensive for us.

 package in.learnjavaskills.mockitotutorial.repository;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;

 public class TransactionRepository
 {
    public boolean isUniqueTransactionId(long transactionId)
    {
        return false;
    }

    public void saveTransaction(TransactionDetail transactionDetail)
    {
        System.out.println("transaction saved");
    }
 }

TransactionDetail and TransactionStatus record

The TransactionDetail and TransactionStatus records are used to accept the request and respond to it.

I'm building an enterprise-level course to cover all topic in the mocktio. We will learn how to check the response and how requests go from one class to another by utilizing these two records, which we will examine later.

 package in.learnjavaskills.mockitotutorial.dto;

 import java.math.BigDecimal;

 public record TransactionDetail(long transactionId, long cardNumber, byte cardExpiryYear,
                                byte cardExpiryMonth, BigDecimal transactionAmount)
 {}

 package in.learnjavaskills.mockitotutorial.dto;

 public record TransactionStatus(short statusCode, String message)
 {}

If you're new to the Java record, I recommend reading What Is a Record in Java?

TransactionException class

This session will go over the in-depth mockito framework, therefore I'd like to demonstrate how quickly we'll be able to identify whether exceptions are raised.

 package in.learnjavaskills.mockitotutorial.exception;

 public class TransactionException extends RuntimeException
 {
    public TransactionException(String message)
    {
        super(message);
    }
 }

Mock the behavior using the given() and willReturn() methods.

Let's start by mock the transactionRepository class, and then we'll limit the calls to isUniqueTransactionId of the transactionRepository by utilizing the BDDMockito class's given() and willReturn() methods.

The transactionRepository class's isUniqueTransactionId method returns false, and that function must return true in order to test the positive case in the creditCardTransaction method.

Therefore, I mocked the transactionRepository class and mocked the isUniqueTransactionId method response with the value true.

To cover the test scenario, the following code from the TransactionService class's creditCardTransaction method must be mocked..

 public boolean isUniqueTransactionId(long transactionId)
 {
     return false;
 }

 boolean uniqueTransactionId = transactionRepository.isUniqueTransactionId(transactionDetail.transactionId());

Here comes the test case, in which we mock the objects and prevent the actual object's calling by passing our own values to the function.

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.dto.TransactionStatus;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.ArgumentMatcher;
 import org.mockito.BDDMockito;
 import org.mockito.InjectMocks;
 import org.mockito.Mock;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 import static org.junit.jupiter.api.Assertions.*;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

    @Test
    void creditCardTransactionPositiveFlow()
    {
        BDDMockito.given(transactionRepository.isUniqueTransactionId(transactionId))
                .willReturn(true);

        TransactionStatus transactionStatus = transactionService.creditCardTransaction(transactionDetail);
        short statusCode = transactionStatus.statusCode();
        assertEquals(statusCode, 200);
    }
 }

The when() and thenReturn() methods can also be used to mock the behavior. The syntax is illustrated below. given() and willReturn() are frequently preferred over when() and thenReturn() since they seem more BDD.

 BDDMockito.when(transactionRepository.isUniqueTransactionId(transactionId))
	.thenReturn(true);

What exactly is ArgumentMatchers?

ArgumentMatchers is a class found in the org.mockito package. ArgumentMatchers provide flexible verification and stubbing of mock behaviors.

If you look closely at our previous example, when we covered mocking the behavior of the TransactionRepository's isUniqueTransactionId function, where we specifically specify the transactionId, you'll notice that

 BDDMockito.given(transactionRepository.isUniqueTransactionId(ArgumentMatchers.anyLong()))
	.willReturn(true);

We may stub it instead of supplying the precise transactionId by just utilizing the ArgumentMatchers class.

 @Test
 void creditCardTransactionPositiveFlow()
 {
	BDDMockito.given(transactionRepository.isUniqueTransactionId(transactionId))
    	.willReturn(true);

	TransactionStatus transactionStatus = transactionService.creditCardTransaction(transactionDetail);
	short statusCode = transactionStatus.statusCode();
    assertEquals(statusCode, 200);
 }

ArgumentMatchers have several advantages

1. We ensure neat, clean, and maintainable test cases by utilizing ArgumentMatchers.

2. ArgumentMatchers enable us to apply BDD by removing hard coded values.

Verify the behavior with the verify() method.

The BDDMockito's verify() method is used to validate behavior, such as whether a specific method is called at least once or not, and many more.

The BDDMockito class has two verify methods: the first takes two parameters, T mock and VerificationMode, while the second accepts just T mock arguments and, by default, considers testing the mock object to be executed only once.

 public static <T> T verify(T mock, VerificationMode mode)

 public static <T> T verify(T mock)

Let's look at an example where we can check whether a transaction is being saved into the database by seeing if the saveTransaction() method of the TransactionRepository class has been called once or not.

  package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.dto.TransactionStatus;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 import static org.junit.jupiter.api.Assertions.*;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

    
    @Test
    void verifyTransactionSaved()
    {
        BDDMockito.given(transactionRepository.isUniqueTransactionId(ArgumentMatchers.anyLong()))
                .willReturn(true);

        transactionService.creditCardTransaction(transactionDetail);

        BDDMockito.verify(transactionRepository, Mockito.times(1))
                .saveTransaction(ArgumentMatchers.any());
    }
 }

VerificationMode

The VerificationMode is nothing except an interface that allows us to check particular behaviors, such as at least once, exactly once, never, and so on.

The org.mockito.verification package contains the VerificationMode interface.

You may use the table below to learn how to validate particular behaviors.

VerificationMode	Explanation
`verify(T mock, Mockito.times(1)`	The times() method accepts one argument, i.e., wantedNumberOfInvocations of type int. This function is used to make sure certain types of behavior happen.
`verify(T mock, Mockito.never())`	As the method name implies, never() method is used to ensure that the specific behavior does not occur at all.
`verify(T mock, Mockito.atLeastOnce())`	Use the atLeastOnce() method to ensure that the activity happened just once.
`verify(T mock, Mockito.atLeast(5)`	The atLeast() method takes one int a parameter, minNumberOfInvocations. We may use this method to ensure that the behavior occurs at least a certain number of times.
`verify(T mock, Mockito.only())`	The only() method may be used to check only one behavior that occurred throughout your test case.
`verify(T mock, Mockito.atMostOnce())`	To ensure that maximum behavior occurs just once, utilize the atMostOnce() method.
`verify(T mock, Mockito.atMost(6))`	The atMost() method, like the atLeast() function, needs one argument of type int, i.e. maxNumberOfInvocations, to evaluate the behavior of the maximum number of invocations.

What exactly is ArgumentCaptor

In order to confirm whether the right values are supplied to the particular method or not, we can capture the argument values using the ArgumentCaptor class in the org.mockito package.

There are two approaches available to us in the mockito framework for capturing the argument: the @Captor annotation or the ArgumentCaptor.forClass(T class). The second method is being used in the example below to capture the argument of the saveTransaction() method.

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.dto.TransactionStatus;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 import static org.junit.jupiter.api.Assertions.*;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

    @Test
    void verifyTransactionSavedWithCorrectValues()
    {
        BDDMockito.given(transactionRepository.isUniqueTransactionId(ArgumentMatchers.anyLong()))
                .willReturn(true);

        transactionService.creditCardTransaction(transactionDetail);

        ArgumentCaptor<TransactionDetail> transactionDetailArgumentCaptor = ArgumentCaptor.forClass(TransactionDetail.class);

        BDDMockito.verify(transactionRepository)
                .saveTransaction(transactionDetailArgumentCaptor.capture());

        TransactionDetail expectedTransactionDetail = transactionDetailArgumentCaptor.getValue();
        assertEquals(transactionDetail.transactionId(), expectedTransactionDetail.transactionId());
        assertEquals(transactionDetail.transactionAmount(), expectedTransactionDetail.transactionAmount());
    }

 }

The argument passed to the specific method can be captured with the assistance of the capture() method, and the values that have been captured can then be retrieved using getValue().

Utilize the @Captor annotation to capture the values of the arguments

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.dto.TransactionStatus;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 import java.math.BigDecimal;

 import static org.junit.jupiter.api.Assertions.*;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @Captor
    ArgumentCaptor<TransactionDetail> transactionDetailArgumentCaptor;

    @InjectMocks
    private TransactionService transactionService;

    private long transactionId = 101L;
    private TransactionDetail transactionDetail = new TransactionDetail(transactionId,
            1234_5678_9101_1213L, (byte) 23, (byte) 11, BigDecimal.TEN);

    @Test
    void verifyTransactionSavedWithCorrectValues()
    {
        BDDMockito.given(transactionRepository.isUniqueTransactionId(ArgumentMatchers.anyLong()))
                .willReturn(true);

        transactionService.creditCardTransaction(transactionDetail);

        BDDMockito.verify(transactionRepository)
                .saveTransaction(transactionDetailArgumentCaptor.capture());

        TransactionDetail expectedTransactionDetail = transactionDetailArgumentCaptor.getValue();
        assertEquals(transactionDetail.transactionId(), expectedTransactionDetail.transactionId());
        assertEquals(transactionDetail.transactionAmount(), expectedTransactionDetail.transactionAmount());
    }
 }

A few practical methods from the ArgumentCaptor class

ArgumentCaptor's method	Description
`T capture()`	The parameter values are captured using Captor(), which is required for verification. This method internally registers a customized ArgumentMatcher implementation. The argument value is retained by this argument matcher for use when making assertions at a later time..
`T getValue()`	The argument's captured value is returned by the getValue() method. If the verified method was called more than once, it would return the most recent captured value.
`List<T> getAllValues()`	The getAllValues() function retrieves all of the values that have been captured when the verified method was called more than once.
`<U, S extends U> ArgumentCaptor<U> forClass(Class<S> clazz)`	Create a new ArgumentCaptor.

Verify to determine if the exception were thrown

If you've noticed, the creditCardTransaction() method of the TransactionService class throws a TransactionException if the method parameter is null.

 if (Objects.isNull(transactionDetail)) 
	throw new TransactionException("transactionDetails must be non null");

It's recommended to test your code to determine if it handles exceptions appropriately.

To utilize Assertions.assertThatThrownBy, I used the assertj-core library. The Gradle dependencies are listed below. here.
testImplementation 'org.assertj:assertj-core:3.24.2' (alert-success)

 package in.learnjavaskills.mockitotutorial.service;

 import in.learnjavaskills.mockitotutorial.dto.TransactionDetail;
 import in.learnjavaskills.mockitotutorial.exception.TransactionException;
 import in.learnjavaskills.mockitotutorial.repository.TransactionRepository;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.mockito.*;
 import org.mockito.junit.jupiter.MockitoExtension;

 @ExtendWith(MockitoExtension.class)
 class TransactionServiceTest
 {
    @Mock
    private TransactionRepository transactionRepository;

    @InjectMocks
    private TransactionService transactionService;

    @Test
    void verifyExceptionThrown()
    {
        Assertions.assertThatThrownBy(()-> { transactionService.creditCardTransaction(null); })
                .hasMessageContaining("transactionDetails must be non null")
                .isExactlyInstanceOf(TransactionException.class);
    }
 }

Conclussion

If you want to be a great developer, you must know how to test your code. In this in-depth mockito framework, we learn how to test your code by utilizing test-driven and behavior-driven development using the mockito and JUnit 5 frameworks.

We've gone over many of the mockito framework's capabilities, such as stubbing with the given() and willReturn() methods to test the code's behavior. We then proceeded to the ArgumentMatchers and ArgumentCaptor.

We covered the various methods for verifying method invocations and the verification modes; we distinguished them and knew which verification mode should be used and when.

Finally, we learn how to ensure that excaptions are handled correctly.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

(getButton) #text=(Next: How to Mock Void Methods with Mockito) #icon=(link) #color=(#2339bd)

Building Lambda Functions with Core Java: A Guide to AWS Lambda Deployment

2025-01-11T16:15:00.003+05:30

AWS Lambda is a serverless compute service offered by Amazon Web Services (AWS). It enables developers to run code without provisioning or managing servers.

Lambda executes code in response to events and automatically manages the computing resources required by that code. This eliminates the need to worry about server maintenance, scaling, and capacity provisioning.

In this tutorial, we will guide you through building your first AWS Lambda function using Core Java and deploying it to AWS Lambda.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Project Dependencies

aws-lambda-java-core is the essential dependency for executing your first AWS Lambda function using Core Java. While this is mandatory, you may also utilize optional dependencies when integrating with other AWS services.

Maven

 <dependencies>
    <!-- Defines handler method interfaces and the context object that the runtime passes to the handler.
    If you define your own input types, this is the only library that you need. -->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-lambda-java-core</artifactId>
        <version>1.2.3</version>
    </dependency>

    <!-- Optional - Input types for events from aws services that invoke Lambda functions. -->
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-lambda-java-events</artifactId>
        <version>3.14.0</version>
    </dependency>
 </dependencies>

Create a deployable JAR file for AWS Lambda using the Maven Shade Plugin, including all necessary built-in libraries.

 <!-- building bundle jar to deploy in aws lambda -->
 <build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.3.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                     <configuration>
                        <createDependencyReducedPom>false</createDependencyReducedPom>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
 </build>

Gradle

 dependencies {
    // Defines handler method interfaces and the context object that the runtime passes to the handler.
    //  If you define your own input types, this is the only library that you need.
    implementation 'com.amazonaws:aws-lambda-java-core:1.2.3'

    // Optional - Input types for events from aws services that invoke Lambda functions.
    implementation 'com.amazonaws:aws-lambda-java-events:3.14.0'
 }

 // building bundle jar to deploy in aws lambda
 jar {
    from { configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) } }
 }

Designing Request and Response Objects for AWS Lambda Handlers

This tutorial requires Java 17 or later. From Java 17 onwards, you can effectively create Java classes or records to elegantly handle request payloads (Lambda events) and responses

For simplicity, I will create two Java records: one for the request and another for the response, each containing relevant parameters.

Request Payload (Lambda Event)

 package in.learnjavaskills;

 public record Request(Integer id, String message) {}

Response

 package in.learnjavaskills;

 public record Response(Integer statusCode, String message) {}

Configuring Lambda Handlers in Java

Let's begin by creating a Java class that implements the RequestHandler interface and overrides the handleRequest() abstract method.

The AWS Lambda function limits us to completing the execution of the entire function within 15 minutes, which implies we can only perform a function execution within 15 minutes.(alert-warning)

 package in.learnjavaskills;

 import com.amazonaws.services.lambda.runtime.Context;
 import com.amazonaws.services.lambda.runtime.LambdaLogger;
 import com.amazonaws.services.lambda.runtime.RequestHandler;

 public class LambdaHandler implements RequestHandler<Request, Response> {
    /**
     * Accept a Request object containing two parameters: an integer 'id' and a string 'message'.
     * Log the received request and then return a 200 status response with a success message.
     */
    @Override
    public Response handleRequest(Request request, Context context) {
        LambdaLogger logger = context.getLogger();
        logger.log("Lambda handler request function invoked with request : " + request.toString());
        return new Response(200, "Success");
    }
 }

• handleRequest() is the main method where your Lambda function's logic resides.

• Context provides information about the invocation, function, and execution environment(e.g., remaining time, log stream name, etc.).

• The RequestHandler<I, O> interface is the cornerstone of developing AWS Lambda functions in Java.

It serves as a contract for your Lambda function's execution logic. By implementing this interface, you tell the Lambda runtime how to handle incoming requests and generate responses.

<I, O> : These represent the input (request) and output (response) types of your Lambda function. You define these based on the specific data your function expects and produces.(alert-passed)

Creating and Deploying Lambda Functions with the AWS Management Console

Navigate to the AWS Management Console at (getButton) #text=(Amazon Console) #icon=(link) #color=(#d35400) Sign in with your AWS credentials if you haven't already. Search for 'Lambda' and click on 'Create function'.

Select the 'Author from scratch' option, provide an appropriate name for your AWS Lambda function (e.g., 'core-java-aws-lambda-function'), and then click the 'Create function' button.

Upon successful creation of your Lambda function, you will see the following success message.

Upload your bundled JAR file. Scroll down to the 'Code' section and select 'Upload from' dropdown button. Then, choose '.zip or .jar file' since you have your bundled JAR.

The AWS Lambda function allows us to upload a jar with a maximum size of 50 MB, and if the jar size is larger than or equivalent to 10 MB, we should use AWS S3.(alert-success)

An upload window will appear. Click the 'Upload' button (as shown in the screenshot below) and select your bundled JAR file from your local machine. Finally, click the 'Save' button.

Next, let's configure the Runtime Settings. Here, you can update the Java version and specify the request handler method path. Click the 'Edit' button to proceed.

In the 'Handler' field, specify in.learnjavaskills.LambdaHandler::handleRequest. You must update this value to match your package, class name, and method name (e.g., your.package.ClassName::yourMethod). You can also configure the Java runtime version, which is set to Java 21 in this example.

Finally, test your Lambda function. Navigate to the Test section and create a new test event. Give it a name (e.g., "test-function") and provide the Event JSON.

This JSON represents the request payload for your function. In our example, we created a Request Java record with an integer 'id' and a string 'message'.

Therefore, the Event JSON should include these two parameters, which your Lambda function expects to receive."

Upon clicking the 'Test' button, you will receive the response from your Lambda function. In this case, you should see a 200 status code and a 'success' message.

Once you've finished your lab exercise, remember to delete your AWS lambda function, layer, and API Gateway.(alert-error)

Conclussion

This tutorial demonstrates how to develop AWS Lambda functions using Core Java and deploy them to the AWS Lambda service. To minimize dependencies, leverage the Lambda Layer functionality to deploy a command library.

AWS Lambda functions have a 15-minute maximum execution time limit.

While direct CPU size configuration is not possible, increasing the allocated memory will proportionally increase the available CPU power

Learn how to build serverless functions with (getButton) #text=(Spring Cloud Function) #icon=(link) #color=(#35a576), seamlessly deploy them on AWS Lambda, and create RESTful APIs using AWS API Gateway in this comprehensive blog.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

How to use Pagination and Sorting with Spring Data JPA | Learn Java Skills

2025-01-04T16:01:00.000+05:30

When retrieving huge amounts of data from a SQL database, the lazy operation is preferable over the eager operation.

You may be required to retrieve a huge amount of data from a SQL database and give it back to the client, and you may be wondering what the performance cost would be if you query a SQL database to retrieve all the data at once.

Yes, you are correct; it is not best practice to query a significant amount of the database to find all the data at once. You may be wondering, what is the ideal approach if the business requirement is to send the entire table of data?

In such scenarios, you have to integrate the concept of pagination or sorting into your application, because even if you return a significant amount of data to your client, it is assumed that your service's users will not use all of the data.

Consider the following e-commerce website: An e-commerce website's backend may have a large quantity of data, yet it only displays a small bit of data at a time. So, if you wish to see additional products, then only they will bring up the next set of products on your screen.

This is what pagination is all about. Fetch the data on demand.

~~toc~~

Dependencies

To use the Spring data JPA in your Spring Boot application, you only need two dependencies. The first is spring-boot-starter-data-jpa, and the second one is about the database you are using. I've utilized MySql throughout this lesson.

Here are more details on Spring Data JPA.

 <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
 </dependency>

 <dependency>
	<groupId>com.mysql</groupId>
	<artifactId>mysql-connector-j</artifactId>
	<scope>runtime</scope>
 </dependency>

Create an entity as well as the repository.

I'm expecting you to have finished configuring your database in your spring boot application. If not, I recommend reading this page to learn how to configure the MySQL database in the spring boot application.

In my MySQL database, I have a table called employee that I'll use to demonstrate pagination and sorting.

I'll make an entity and a repository for this table.

Employee Entity

 package in.learnjavaskills.paginationandsorting.entity;

 import jakarta.persistence.*;

 import java.util.Objects;

 @Entity
 @Table(name = "employee")
 public class Employee
 {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Column(name ="id")
    private Long id;

    @Column(name = "first_name", nullable = false)
    private String firstName;

    @Column(name = "last_name")
    private String lastName;

    @Column(name = "address")
    private String address;
    // getter and setter, constructor, equal and hashcode, tostring
 }

Employee Repository

 package in.learnjavaskills.paginationandsorting.repository;

 import in.learnjavaskills.paginationandsorting.entity.Employee;
 import org.springframework.data.repository.PagingAndSortingRepository;

 public interface EmployeeRepository extends PagingAndSortingRepository<Employee, Long>
 {}

Here I used the PagingAndSortingRepository interface; you can also use the JpaRepository interface.

The JpaRepository interface extends the ListPagingAndSortingRepository interface and the ListPagingAndSortingRepository interface extends the PagingAndSortingRepository interface.

Pagination with the spring data JPA in the spring boot

Pagination is utilized when we wish to fetch all the data from the database, as was covered earlier in this lesson. We fetch the data chunk by chunk using the pagination idea to enhance performance.

Pageable Interface

The pageable is an interface that offers a number of pagination-related operations.Some of the most widely used methods are as follows:

Method's	Description
`int getPageSize();`	Use the getPageSize method to determine how many pages there are in total
`int getPageNumber();`	You can obtain the current page by using the getPageNumber method.
`Sort getSort();`	The getSort method is used for finding the Sort class.
`Pageable next();`	Use the next method to move to the next Pageable.
`Pageable previousOrFirst();`	Use the previousOrFirst function to go to the previous or first Pageable.
`Pageable first();`	The first method is availabe to jump to the first Pageable.
`boolean hasPrevious();`	To determine whether the previous pageable is available or not, use the hasPrevious method.
`Pageable withPage(int pageNumber)`	By specifying the page number using the withPage method, you can expect the exact pageable.

How do I create an instance of pageable?

To build a pageable instance, we can use the static methods of the PageRequest class.

Let's execute our first pagination program to better understand its operation.

In this example, I'll use the PagingAndSortingRepository interface's findAll method to retrieve the whole set of records from the employee table.

 @Test
 void findAllData()
 {
   Pageable pageable = PageRequest.of(0, 1);
   Page<Employee> page = employeeRepository.findAll(pageable);
   System.out.println("total pages : " + page.getTotalPages());
   if (page.hasContent())
   {
 	 List<Employee> employeeList = page.getContent();
 	 if (Objects.nonNull(employeeList) && !employeeList.isEmpty())
 	   employeeList.forEach(System.out :: println);
   }
 }

As you can see from the example above, we have a repository instance called employeeRepository, and we performed the pagination using the findAll method , which accepts a pageable object.

PageRequest.of() method takes two arguments, which are as follows:
pageNumber: This is a parameter of the int type, and as its name suggests, you must supply it with the page number for which you want to retrieve page data.
pageSize: This option, which is of the int type, asks you to specify how many records to show each page.
To create an instance of the Pageable interface, use the PageRequest.of() function.
(alert-passed)

Sorting with spring data JPA in the spring boot

The Sort class, which is part of the org.springframework.data.domain package, can be used to sort the record.

You can sort your results using any of the numerous overloaded Sort.by() methods that are available, depending on your needs.

As you can see, I sorted the outcome by the employee's first name in ascending order using the sort.by() method.

 @Test
 void sortData()
 {
   Sort sortByFirstNameInAscOrder = Sort.by(Sort.Direction.ASC, "firstName");
   Pageable pageable = PageRequest.of(0, 4, sortByFirstNameInAscOrder);
   Page<Employee> page = employeeRepository.findAll(pageable);
   System.out.println("total pages : " + page.getTotalPages());
   if (page.hasContent())
   {
     List<Employee> employeeList = page.getContent();
     if (Objects.nonNull(employeeList) && !employeeList.isEmpty())
       employeeList.forEach(System.out :: println);
   }
}

Here, I used the Sort by(Direction direction, String... properties) method with the following two arguments:
Direction: The Sort class contains an enum called "Direction." We can use it to specify the sort order we desire, such as ASC for ascending and DESC for descending.
Arrays of String: which accepts the field's name for sorting; in this case, we used the employee's first name.(alert-success)

Sort by(Direction direction, String... properties)~~(code-box)~~

Is sorting the direction possible solely using the Direction enum? In addition, you may adjust the sorting direction by using the Sort class's ascending() or descending() method.(alert-passed)

Sort sortByFirstNameInAscOrder = Sort.by("firstName").ascending();
Sort sortByFirstNameInDesOrder = Sort.by("firstName").descending();~~(code-box)~~

Is there the ideal way to use sorting without utilizing pagination?

We saw how to sort the result with pagination in the previous example, but are we able to sort the result without using pagination in our application?

Yes, you are correct; we can sort the result in spring data JPA without requiring pagination. We will have the similar experience in the following session.

 @Test
 void sortDataWithoutPagination()
 {
   Sort sortByFirstNameInDesOrder = Sort.by("firstName").descending();
   Pageable pageable = PageRequest.of(0, 4, sortByFirstNameInDesOrder);
   List<Employee> employeeList = (List<Employee>) employeeRepository.findAll(sortByFirstNameInDesOrder);

   if (Objects.nonNull(employeeList) && !employeeList.isEmpty())
	 employeeList.forEach(System.out :: println);
 }

How to use pagination and sorting on the user-defined query

We had previously used the pagination and sorting on the Spring data JPA predefine method, which were provided out of the box.

If you've come this far, you must have wondered about it. It's OK; we can utilize the pagination and sorting on the Spring data JPA's interface function, but what if we need to execute the same pagination and sorting on our own query?

The following section will walk us through how we will be able to do pagination and sorting on the user-defined query.

To allow pagination in your custom query, accept the pageable interface in the abstract method of your repository.

I'm implementing the JPQL query right here in the employee repository. The native query will also work for both queries, or you are able to follow to the exact same process.

You can also learn how to use the DTO project in spring boot with spring data JPA by clicking here.

 package in.learnjavaskills.paginationandsorting.repository;

 import in.learnjavaskills.paginationandsorting.entity.Employee;
 import org.springframework.data.domain.Page;
 import org.springframework.data.domain.Pageable;
 import org.springframework.data.jpa.repository.Query;
 import org.springframework.data.repository.PagingAndSortingRepository;

 public interface EmployeeRepository extends PagingAndSortingRepository<Employee, Long>
 {
    @Query("from Employee")
    Page<Employee> findAllEmployee(Pageable pageable);
 }

Once you've finished declaring the abstract method in the repository layer with the pageable interface, you may utilize it exactly as we did earlier in this article in the service layer.

 @Test
 void paginationOnUserDefineMethod()
 {
   Sort sortByFirstNameInDesOrder = Sort.by("firstName").descending();
   Pageable pageable = PageRequest.of(0, 4, sortByFirstNameInDesOrder);
   Page<Employee> page = employeeRepository.findAllEmployee(pageable);
   System.out.println("total pages : " + page.getTotalPages());
   if (page.hasContent())
   {
     List<Employee> employeeList = page.getContent();
     if (Objects.nonNull(employeeList) && !employeeList.isEmpty())
       employeeList.forEach(System.out :: println);
   }
 }

What exactly is the Slice interface? In Spring Data JPA pagination, how do I use the Slice interface?

Slice is an interface in the org.springframework.data.domain package that allows us to get a single slice of data. We've learnt that if we want to use pagination, we can use the page interface as the return type of the result, but you can also use the slice interface instead.

In the Page interface, we must explicitly specify the page number to retrieve the data, whereas in the Slice interface, we simply utilize the hasNext() method to detect whether the next batch is available or not.

You can access the next batch of data using the nextPageable() method once the next batch is available.

In the following part, we'll look at how the slice interface interacts with Pageable.

 package in.learnjavaskills.paginationandsorting.repository;

 import in.learnjavaskills.paginationandsorting.entity.Employee;
 import org.springframework.data.domain.Pageable;
 import org.springframework.data.domain.Slice;
 import org.springframework.data.jpa.repository.Query;
 import org.springframework.data.repository.PagingAndSortingRepository;

 public interface EmployeeRepository extends PagingAndSortingRepository<Employee, Long>
 {
    @Query("from Employee")
    Slice<Employee> findAllEmployee(Pageable pageable);
 }

Let's look at how the slice interface can be used in the service layer. As previously explained, we must utilize the hasNext() method to verify whether the next batch is accessible and then use nextPageable() to move on to the next batch.

 @Test
 void paginationOnUsingTheSlice()
 {
  Sort sortByFirstNameInDesOrder = Sort.by("firstName").descending();
  Pageable pageable = PageRequest.of(0, 1, sortByFirstNameInDesOrder);
  Slice<Employee> slice = employeeRepository.findAllEmployee(pageable);

  if (slice.hasContent())
  {
    List<Employee> employeeList = slice.getContent();
    if (Objects.nonNull(employeeList) && !employeeList.isEmpty())
      employeeList.forEach(System.out :: println);

    while (slice.hasNext())
    {
      slice = employeeRepository.findAllEmployee(slice.nextPageable());
      employeeList = slice.getContent();
      if (Objects.nonNull(employeeList) && !employeeList.isEmpty())
        employeeList.forEach(System.out :: println);
    }
  }
 }

Slice is an interface in the org.springframework.data.domain package that allows us to get a single slice of data.
The hasContent() method determines whether or not the batch of data is empty.
The hasNext() method is used to see whether the next bacth is available.
The nextPageable() method is used to get the next pageable interface instance in order to get the next batch data.(alert-passed)

What is the key difference between the page interface and the slice interface?

The org.springframework.data.domain package contains both the page interface and the slice interface.

The page interface extends the slice interface, which means that it has all of the capabilities of the slice interface, whereas the slice interface extends the streamable interface.

The page interface includes two additional methods, such as the
getTotalPages(): gets the total number of pages accessible.
getTotalElements(): The total number of elements is returned by getTotalElements().(alert-passed)

The page interface use case

When you wish to display the number of pages available in the user interface (UI), utilize the Page interface. Because you may use the getTotalPages() function to display the number of pages.

for example, at the bottom of the e-commerce website, display the number of pages accessible to browse to the next or previous page or directly jump to the user's chosen page number.

The Slice interface use case

Use the Slice interface when you don't want to show the amount of records in the initial query or when you wish to fetch data on demand, such as on a lazy loading website, such as a social media site, where content is loaded while scrolling or hitting the load button.

Conclussion

We covered how to implement pagination in a Spring Boot application using Spring Data JPA in this instructional article.

We've seen the spring data JPA's default method, which includes pagination by default; however, we've also utilized pagination in a user-defined method or query.

In this post, you will not only learn about pagination, but also about sorting with the Sort class.

In the pagination concept, we learned about the various return type interfaces, such as the page interface and the slice interface, and we were able to differentiate between the two.

In addition, we go over the use cases for both the page interface and the slice interface in depth.

Keep learning and keep growing.

(getButton) #text=(Next: Spring Data JPA DTO Projections) #icon=(link) #color=(#2339bd)

Spring Cloud AWS & SQS: Streamline Your Development Workflow with Reliable Queuing | Learn Java Skills

2024-12-27T15:31:00.000+05:30

Building modern, scalable applications sometimes need effective communication across many components. However, typical synchronous communication can cause bottlenecks and reduce performance.

Asynchronous messaging enables programs to exchange messages without waiting for a response, resulting in greater responsiveness and robustness.

This blog article explains how to use Spring Cloud AWS and AWS SQS to establish strong asynchronous communication in your microservices architecture.

We'll walk you through integrating SQS with your Spring Boot application, sending and receiving messages, and dealing with any issues. By the conclusion, you'll have the knowledge and skills you need to improve communication inside your apps and grasp the benefits of an asynchronous approach.

~~toc~~

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Amazon Simple Queue Service (SQS)

Amazon SQS offers queues for high-volume, system-to-system communications. Queues may be used to decouple heavyweight processes as well as buffer and batch operations.

Amazon SQS retains messages until microservices and serverless applications can process them.

Amazon SQS enables producers to put messages into a queue. The messages are then saved in a SQS Queue. When consumers are ready to process new messages, they retrieve them from the queue.

Applications, microservices, and numerous AWS services can function as either producers or consumers.

Spring Cloud AWS

Spring Cloud for Amazon Web Services is an extension of the Spring Cloud umbrella project, which enables Spring Boot applications to easily communicate with Amazon Web Services.

Spring Cloud AWS provides an effortless way for accessing AWS services through well-known Spring idioms and APIs, such as the messaging or caching API.

Developers may design apps based on AWS hosted services without having to worry about infrastructure or maintenance.

Creating Your First Message Queue: A Step-by-Step Guide to Amazon SQS

Access the AWS Management Console:

Navigate to the AWS Management Console at (getButton) #text=(Amazon Console) #icon=(link) #color=(#d35400) Sign in with your AWS credentials if you haven't already.

Open the Amazon SQS service:

Search for SQS in the search box or find it in the Messaging category of the service list.

Create a new queue:

Click on the Create queue button.

Choose an appropriate name for your queue. In my situation, I named the my-first queue. Amazon SQS offers two types of queues: Standard and FIFO. For this example, I'll be creating a standard queue.

You can leave the default settings for most use cases or customize them based on your specific needs (e.g., visibility timeout, message retention period).

Configure the access policy to determine who may send and receive messages from this queue. I created an IAM user, sqs-queue-read-write, to send and receive messages programmatically. Must provide the ARN of the IAM user here.

Click on Create queue to finalize the process.

Spring Cloud AWS and SQS: Essential Dependencies for Streamlining Your Messaging

spring-cloud-aws-starter-sqs: Provides messaging abstractions for sending and receiving messages.

 // spring cloud aws BOM(Bill of materials)
 implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
 // Amazon SQS 
 implementation 'io.awspring.cloud:spring-cloud-aws-starter-sqs'

Configuring Amazon SQS in Spring Cloud AWS

Securely Configuring AWS Credentials: Different Approaches

Spring Cloud AWS starter automatically configures a DefaultCredentialsProvider that checks for AWS credentials in the following order:

• Java System Properties: aws.accessKeyId, aws.secretAccessKey.

• Environment variables include AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

• Credentials for Web Identity Token can be obtained from system configurations or environmental variables.

• Credential profiles are stored in the default directory (~/.aws/credentials) used by all AWS SDKs and the AWS CLI.

• Credentials provided through the Amazon EC2 container service if the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is defined and the security manager has access to the variable.

• Instance profile credentials are given via the Amazon EC2 metadata service.

Configuring SQS Properties in Application.properties

 # Amazon SQS configuration
spring.cloud.aws.credentials.access-key=${AWS_ACCESS_KEY_ID}
spring.cloud.aws.credentials.secret-key=${AWS_SECRET_ACCESS_KEY}
spring.cloud.aws.region.static=${AWS_REGION}
spring.cloud.aws.sqs.endpoint=${SQS_ENDPOINT}

Granting IAM User Access to an SQS Queue

Securely grant your IAM user the required permissions to interact with the SQS queue within your account.

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "sqs-read-and-write-policy",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:GetQueueAttributes"
            ],
            "Resource": "your sqs arn e.g: arn:aws:sqs:*:123456789012:*"
        }
    ]
 }

Sending a Message to an Amazon SQS Queue

Spring Cloud AWS offers the SqsTemplate for sending messages to the Amazon SQS.

Using SqsTemplate for SQS Messaging

When leveraging Spring Boot and autoconfiguration, a SqsTemplate instance is automatically autowired if no other template bean is detected in the context.

This template instance is supported by the autoconfigured SqsAsyncClient, which accepts any configurations given. SqsTemplates are immutable and thread-safe.

If more extensive setup is preferred, a builder is also provided along with a set of options.

 package in.learnjavaskills.sqsmessaging.configuration;

 import io.awspring.cloud.sqs.operations.SqsTemplate;
 import io.awspring.cloud.sqs.operations.TemplateAcknowledgementMode;
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;
 import software.amazon.awssdk.services.sqs.SqsAsyncClient;

 import java.time.Duration;

 @Configuration
 public class SqsTemplateConfiguration
 {
    @Bean
    public SqsTemplate sqsTemplate(SqsAsyncClient sqsAsyncClient)
    {
        return SqsTemplate.builder()
                .sqsAsyncClient(sqsAsyncClient)
                .configure(options -> options
                        .acknowledgementMode(TemplateAcknowledgementMode.MANUAL)
                        .defaultQueue("my-first-queue")
                        .defaultPollTimeout(Duration.ofSeconds(5))
                        .defaultMaxNumberOfMessages(5)
                )
                .build();
    }
 }

Sending Messages with SqsTemplate

There are several methods for sending messages to SQS queues using the SqsTemplate. The SqsOperations interface provides the following methods, which have async equivalents in SqsAsyncOperations.

Method	Description
`SendResult<T> send(String queue, T payload);`	Send a message to the provided queue with the given payload
`SendResult<T> send(T payload);`	Send a message to the configured default endpoint
`SendResult<T> send(String queue, Message<T> message);`	Send the given Message to the provided queue
`SendResult<T> send(Consumer<SqsSendOptions> to);`	Send a message with the provided options
`SendResult.Batch<T> sendMany(String queue, Collection<Message<T>> messages);`	Send a batch of Messages to the provided queue

An example using the SqsSendOptions variant follows:

 SendResult<Object> sendResult = sqsTemplate.send(to -> to.queue("my-first-queue")
	.payload("Hello,  My first message to amazon SQS")
	.header("header1", "header-value1")
	.delaySeconds(2));

An example of sending a message using a Java class is:

 Employee employee = new Employee("name", "designation", "department");
 SendResult<Object> sendResult = sqsTemplate.send(to ->to.queue("my-first-queue")
	.payload(employee)
	.header("header1", "header-value1")
	.headers(Map.of("My-second-header", "my-second-header-value"))
	.delaySeconds(1));

Understanding the SendResult of SqsTemplate.send

The SendResult record offers helpful information about the sending operation.

 public record SendResult<T>(UUID messageId, String endpoint, Message<T> message, Map<String, Object> additionalInformation) {
    public record Batch<T>(Collection<SendResult<T>> successful, Collection<SendResult.Failed<T>> failed) {}
    public record Failed<T> (String errorMessage, String endpoint, Message<T> message, Map<String, Object> additionalInformation) {}
 }

When the send actions succeeds, the SendResult object is created with:

`messageId`	The messageId returned from SQS for the message
`endpoint`	The endpoint the message was sent to
`Message`	The Message instance that was sent, with any additional headers that might have been added by the framework
`additionalInformation`	An additionalInformation map with the sequenceNumber generated for the message in Fifo queues

When the send operation for a single message fails, a MessagingOperationFailedException is thrown, which contains the message.

For Batch send operations, a SendResult.Batch object is returned. This object includes a collection of successful and failed results.

If any messages failed to be sent inside a batch, return the relevant SendResult.Failed objects are created.

Receiving Messages with SqsTemplate

The SqsTemplate provides handy methods for receiving messages from the Standard and FIFO SQS queues.

These methods are divided into two interfaces that SqsTemplate implements: SqsOperations and SqsAsyncOperations. If just sync or async actions are required, utilizing the specified interface might help cut down the options.

The SqsOperations interface provides the following methods, which have async equivalents in SqsAsyncOperations.

Method	Description
`Optional<Message<?>> receive();`	Receive a message from the configured default endpoint and options
`<T> Optional<Message<T>> receive(String queue, Class<T> payloadClass);`	Receive a message from the provided queue and convert the payload to the provided class
`Optional<Message<?>> receive(Consumer<SqsReceiveOptions> from);`	Receive a message with the provided options
`<T> Optional<Message<T>> receive(Consumer<SqsReceiveOptions> from, Class<T> payloadClass);`	Receive a message with the provided options and convert the payload to the provided class
`Collection<Message<?>> receiveMany();`	Receive a batch of messages from the configured default endpoint and options.
`<T> Collection<Message<T>> receiveMany(String queue, Class<T> payloadClass);`	Receive a batch of messages from the provided queue and convert the payloads to the provided class.
`Collection<Message<?>> receiveMany(Consumer<SqsReceiveOptions> from);`	Receive a batch of messages with the provided options
`<T> Collection<Message<T>> receiveMany(Consumer<SqsReceiveOptions> from, Class<T> payloadClass);`	Receive a batch of messages with the provided options and convert the payloads to the provided class.

The following is an example for receiving a message with options:

 Optional<Message<Employee>> receivedMessage = sqsTemplate.receive(from -> from.queue("my-first-queue")
	.visibilityTimeout(Duration.ofSeconds(10))
	.pollTimeout(Duration.ofSeconds(10)), Employee.class);

The SqsTemplate by default acknowledges all received messages, which can be changed by setting TemplateAcknowledgementMode.MANUAL in the template options:(alert-warning)

 SqsTemplate.builder()
	.configure(options -> 
    	options.acknowledgementMode(TemplateAcknowledgementMode.MANUAL));

Consuming Messages from SQS Queues using @SqsListener

The easiest approach to consume SQS messages is to use the @SqsListener annotation on a method in a @Component class.

The framework will then build the MessageListenerContainer and configure a MessagingMessageListenerAdapter to call the method whenever a message is received.

Defining Queue Names for @SqsListener Annotation (Spring Cloud AWS)

One or more queues can be provided in the annotation using either the queueNames or value attributes; there is no differentiation between the two.

Queue URLs can be used instead of queue names. Using urls instead of queue names might result in significantly faster startup times since it prevents the framework from searching for the queue url when the containers start.

 @SqsListener({"my-first-queue", "${my.second.queue.url}"})
 public void listener(String message) {
    System.out.println("received message : "  + message);
 }

Any number of @SqsListener annotations can be used in a bean class, and each annotated method will be handled by a separate MessageListenerContainer.(alert-success)

Configuring Advanced Options with @SqsListener

The following properties can be supplied using the @SqsListener annotation. Such properties override the corresponding SqsContainerOptions for the generated MessageListenerContainer.

`id`	Indicate the resultant container's id. This may be used to retrieve the container from the MessageListenerContainerRegistry, and it is also utilized by the container and its components for general logging and thread identification.
`maxConcurrentMessages`	Determine the maximum number of messages that can be in flight at any given point in time.
`pollTimeoutSeconds`	Set the maximum amount of time to wait for a poll to return from SQS. Please keep in mind that if there are messages available, the call may return before this time.
`messageVisibilitySeconds`	Set the minimum visibility for messages collected in a poll. Note that for FIFO single message listener methods, this visibility is applied to the whole batch before each message is delivered to the listener.

Here's an example of @SqsListener annotation properties:

 @SqsListener(queueNames = "my-first-queue", id = "my-first-queue-id",
            maxConcurrentMessages = "10", pollTimeoutSeconds = "2",
            messageVisibilitySeconds = "2")
 public void listener(Message<Employee> message) {
   System.out.println("received message : "  + message.getPayload());
 }

Understanding Listener Method Arguments in @SqsListener

The listener method signature allows a variety of argument types, which are as follows:

Listener Method Arguments	Description
`MyPojo`	POJO types are automatically de-serialized from JSON.
`Message<MyPojo>`	Retrieves a Message<MyPojo> object with deserialized payload and MessageHeaders.
`List<MyPojo>`	Enables batch mode and gets the batch polled by SQS.
`List<Message<MyPojo>>`	Enables batch mode and gets the batch polled from SQS, featuring all the headers.
`@Header(String headerName)`	provides the specified header.
`@Headers`	provides the MessageHeaders or a Map<String, Object>
`Acknowledgement`	provides methods for manually acknowledging messages for individual message listeners. AcknowledgementMode must be set to Manual.
`BatchAcknowledgement`	Batch listeners can manually acknowledge partial or entire message batches. AcknowledgementMode must be set to Manual.
`Visibility`	provides the changeTo() method that enables changing the message’s visibility to the provided value.
`QueueAttributes`	Provides queue attributes for the message's receiving queue.
`software.amazon.awssdk.services.sqs.model.Message`	provides the original Message from SQS

Here's an example with multiple arguments:

 @SqsListener("my-first-queue")
 public void listener(Message<Employee> message, MessageHeaders messageHeaders,
		 Visibility visibility, QueueAttributes queueAttributes, Acknowledgement acknowledgement,
		 software.amazon.awssdk.services.sqs.model.Message originalMessage) {
         
	System.out.println("received message : "  + message.getPayload());
	System.out.println("message header : " + messageHeaders.toString());

	acknowledgement.acknowledge();
	visibility.changeTo(10);

	String queueUrl = queueAttributes.getQueueUrl();
	System.out.println("queue url : " + queueUrl);

	System.out.println("original message : " + originalMessage.toString());
 }

Batch listeners support a single List<MyPojo> and List<Message<MyPojo>> method arguments, and an optional BatchAcknowledgement or AsyncBatchAcknowledgement arguments. MessageHeaders should be extracted from the Message instances through the getHeaders() method.(alert-warning)

Handling Failed Messages: Configuring a Dead-Letter Queue for Amazon SQS

Let's delve deeper into robust message handling. This (getButton) #text=(Dead-Letter Queue for Amazon SQS) #icon=(link) #color=(#35a576) post will guide you through configuring a Dead-Letter Queue (DLQ) for your SQS queues, ensuring that failed messages are properly captured and managed for troubleshooting and recovery.

Conclusion: Leveraging SQS with Spring Cloud AWS

In this blog post, we looked at how Amazon SQS and Spring Cloud AWS can work together to provide robust message queuing in your application. We looked into:

• Creating a SQS Queue in the AWS Console: To establish the foundation, we demonstrated how to create your SQS queue.

• Sending and Receiving Messages with SqsTemplate: We showed how to effectively send and retrieve messages using the Spring Cloud AWS SqsTemplate class.

• Using the @SqlListener annotation: We looked into the @SqsListener annotation, which provides a strong method for asynchronous message ingestion from a SQS queue.

By combining these principles, you provide your Spring Cloud AWS application with the capability to:

• Decouple services: Create loose coupling between services by outsourcing communication to an asynchronous message queue.

• Increase scalability and fault tolerance: SQS makes it easier to handle traffic surges and allows individual services to continue processing messages even if they fail.

• Improve message delivery reliability: SQS assures at-least-once delivery, guaranteeing that your messages reach their target.

By incorporating SQS into your Spring Cloud AWS environment, you acquire a powerful tool for creating strong, scalable, and resilient messaging-driven systems. We urge you to experiment and discover how SQS might help your apps communicate and manage messages more efficiently.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

How to configure a Dead-Letter Queue (DLQ) for Amazon SQS

2024-12-27T15:13:00.011+05:30

Dead letter queues (DLQs) are crucial in distributed systems for handling failed message processing. They isolate problematic messages, preventing main queue blockage and enabling investigation, retry attempts, or archiving for later analysis, ensuring no data loss.

This lesson covers enabling and creating dead-letter queues in Amazon SQS.

The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue.(alert-warning)

~~toc~~

Creating and Configuring Dead-Letter Queues using the AWS CLI

The AWS CLI provides a simple method for configuring dead-letter queues (DLQs) in Amazon SQS. The process requires first creating an SQS queue to serve as the DLQ, followed by creating the primary queue and configuring it to send failed messages to the designated DLQ.

Creating a Dead-Letter Queue

 aws sqs create-queue --queue-name dead-letter-queue

After successfully executing the script, you will receive a JSON response containing the URL of the created SQS queue.

 {
    "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/dead-letter-queue"
 }

Creating a Standard SQS Queue with Dead-Letter Queue Configuration and Redrive Policy

Create a JSON file containing the following RedrivePolicy details, which are required to configure a dead-letter queue when creating a new Amazon SQS queue.

 {
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:dead-letter-queue\",\"maxReceiveCount\":\"3\"}",
  "MessageRetentionPeriod": "259200"
 }

To create the new Amazon SQS queue, use the same AWS CLI create-queue command as before, but this time include the --attributes flag and specify the path to the JSON file.

 aws sqs create-queue --queue-name main-queue --attributes file://create-queue.json

 {
    "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/main-queue"
 }

What is a RedrivePolicy in the context of dead-letter queues?

A RedrivePolicy specifies how messages are moved from the main queue to the DLQ after a certain number of failed processing attempts.

Specifically, the RedrivePolicy contains two key components:

• maxReceiveCount : This is the maximum number of times a consumer can attempt to receive and process a message before it is sent to the DLQ. When a consumer gets a message but fails to process it (due to an error, timeout, or explicit rejection), the receive count for that message is increased. When the receive count hits maxReceiveCount, the message is automatically transferred from the main queue to the related DLQ.

• deadLetterTargetArn : This is the Amazon Resource Name (ARN) of the dead-letter queue where the message will be sent after exceeding the maxReceiveCount. This clearly identifies the destination queue for failed messages.

Creating and Configuring Dead-Letter Queues using the Amazon Console

For those without AWS CLI access, the AWS Management Console provides an alternative method for creating and configuring dead-letter queues in Amazon SQS.

Creating a Dead-Letter Queue

In the (getButton) #text=(Amazon Console) #icon=(link) #color=(#d35400), navigate to the SQS service by searching for "SQS," as shown in the following screenshot.

Click "Create queue" to begin creating your first queue. We'll start by creating the dead-letter queue.

We're creating a standard queue called dead-letter-queue. The same process applies to FIFO queues.

Several optional configurations are available, but we will proceed with the default settings for this example.

The queue creation process is finalized by clicking the "Create queue" button located at the bottom of the screen.

Creating a Standard SQS Queue with Dead-Letter Queue Configuration

Let's proceed to create the main queue with the dead-letter queue configuration. Since the dead-letter queue is a standard queue, we will also create a standard main queue in this example.

Enabling the dead-letter queue for this main queue is accomplished by providing the ARN of the designated DLQ.

The queue creation process is finalized by clicking the "Create queue" button located at the bottom of the screen.

Conclusion

You now have the tools to effectively manage message processing errors in your Amazon SQS queues. Whether you prefer the visual interface of the AWS Management Console or the scripting power of the AWS CLI, you can now seamlessly integrate DLQs into your architecture.

This ensures that transient errors don't lead to permanent data loss and allows you to investigate and resolve underlying issues more effectively.

Remember to regularly monitor your DLQs and implement appropriate alerting mechanisms to proactively address any recurring problems.

Keep learning and keep growing.

DynamoDB Transactions with TransactWriteItems and TransactGetItems with Spring Boot | LearnJavaSkills

2024-11-24T20:22:00.000+05:30

Amazon DynamoDB is a fully managed NoSQL database service that offers high performance, scalability, and availability for any application. One of its key features that sets it apart from other NoSQL databases is its support for transactions.

Transactions in DynamoDB are a crucial mechanism for ensuring data consistency and integrity in your applications. They allow you to perform multiple operations on multiple items within a single atomic unit, guaranteeing that either all operations are successful or none of them are.

The two main types of operations supported by DynamoDB transactions are TransactWriteItems and TransactGetItems.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Dependencies

 ext {
	springCloudAwsVersion = '3.0.0'
 }
 
 dependencies {
	// spring cloud aws BOM(Bill of materials)
	implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
	// dynamoDB
	implementation 'io.awspring.cloud:spring-cloud-aws-starter-dynamodb'
 }

Application Configuration (application.properties)

 # DynamoDB connection
 spring.cloud.aws.credentials.access-key=
 spring.cloud.aws.credentials.secret-key=
 # Choose your region according to your table. Because I have a DynamoDB table in the ap-south-1 area, I've set the endpoint and region.
 spring.cloud.aws.region.static=ap-south-1
 spring.cloud.aws.endpoint=https://dynamodb.ap-south-1.amazonaws.com

Want to learn more about efficient data management in DynamoDB? My tutorial on (getButton) #text=(DynamoDB BatchWriteItem and BatchGetItem in Java Spring Boot Applications) #icon=(link) #color=(#35a576) is a great starting point.(alert-passed)

TransactWriteItems API

TransactWriteItems is a synchronous, idempotent operation that allows you to group up to 100 write actions into a single atomic transaction, ensuring either complete success or failure.

These actions can target up to 100 distinct items across multiple tables within the same region.The aggregate size of the items in the transaction cannot exceed 4 MB

Unlike BatchWriteItem operations, TransactWriteItems require all included actions to be completed successfully for the entire transaction to be considered successful. With BatchWriteItem, individual actions can succeed or fail independently.(alert-warning)

You can't target the same item with multiple operations within the same transaction. For example, you can't perform a ConditionCheck and also an Update action on the same item in the same transaction.

software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Transaction request cannot include multiple operations on one item(alert-error)

To illustrate the practical application of all four TransactWriteItems operations, we'll use an e-commerce scenario. I have four tables: customer, orders, products, and cart, and we'll simulate a customer placing an order.

ConditionCheck: Ensuring Data Integrity with Conditional Writes

ConditionCheck Determines if an item exists or meets certain criteria.

When we need to determine if a certain condition is true, it's often best to use condition checking. For example, before allowing a customer to place an order, I'll check if their account is active.

 private ConditionCheck customerExistsConditionCheck() {
        // customer table partition key and sort key
        HashMap<String, AttributeValue> customerKey = new HashMap<>();
        customerKey.put("customer_id", AttributeValue.fromS("c_123")); // partition key
        customerKey.put("email", AttributeValue.fromS("dummy@email.com")); // sort key

        // expression attribute values is fill the expected values in the query placeholder.
        Map<String, AttributeValue> customerExpressionAttributeValue = new HashMap<>();
        customerExpressionAttributeValue.put(":expected_active_status", AttributeValue.fromBool(true));

        // verify attribute 'active' exist in the items and active status must be true.
        String attribute = "active";
        return ConditionCheck.builder()
                .tableName("customer")
                .key(customerKey)
                .conditionExpression("attribute_exists(" + attribute + ")")
                .conditionExpression("active = :expected_active_status")
                .expressionAttributeValues(customerExpressionAttributeValue)
                .build();
    }

tableName() method

The tableName method in the ConditionCheck class of the AWS SDK for Java is primarily used to specify the DynamoDB table that you want to perform the conditional operation on

key() method

The ConditionCheck class's key method accepts a partition key and a sort key as parameters.

conditionExpression() method

The conditionExpression method in the ConditionCheck class of the AWS SDK for Java is used to specify a logical expression that must be satisfied for a DynamoDB operation to succeed.

The conditionExpression is typically combined with withExpressionAttributeValues and withExpressionAttributeNames to define the specific attributes and values to be used in the expression.

expressionAttributeValues() method

The expressionAttributeValues method in the ConditionCheck class of the AWS SDK is used to define the actual values that will be used in the condition expression.

When you create a condition expression, you often use placeholders for attribute values. The expressionAttributeValues method allows you to map these placeholders to their corresponding actual values. This ensures that the condition expression is evaluated correctly with the correct data.

Functions in DynamoDB

attribute_exists: True if the item contains the attribute specified by path.
Example: Check whether an item in the Product table has a side view picture.

attribute_exists (#Pictures.#SideView)

To explore DynamoDB functions in depth, check out Condition and filter expressions, operators, and functions

Put: Adding or Updating Items

Put Initiates a PutItem operation to create a new item or update an existing one, either unconditionally or based on specified conditions

 private Put placeOrder() {
        // create a map with key value pair to insert a record into order table
        Map<String, AttributeValue> orderItems = new HashMap<>();
        orderItems.put("order_id", AttributeValue.fromS("od_123")); // partition key
        orderItems.put("product_id", AttributeValue.fromS("p_123")); // sort key
        orderItems.put("customer_id", AttributeValue.fromS("c_123"));
        orderItems.put("total_amount", AttributeValue.fromS("101"));
        orderItems.put("status", AttributeValue.fromS("CONFIRM"));

        return Put.builder()
                .tableName("orders")
                .item(orderItems)
                // An optional parameter that returns the item attributes for an UpdateItem operation that failed a condition check.
                .returnValuesOnConditionCheckFailure(ReturnValuesOnConditionCheckFailure.ALL_OLD)
                .build();
    }

item() method

item: The items method takes a Map<String, AttributeValue> as an argument. This map represents the attributes and their corresponding values that you want to associate with the item.

• Key: The key in the map represents the attribute name.

• Value: The value is an AttributeValue object that contains the actual value for the attribute. This value can be of various types, such as a string, number, or list.

ReturnValuesOnConditionCheckFailure() method

ReturnValuesOnConditionCheckFailure: The ReturnValuesOnConditionCheckFailure parameter determines what DynamoDB should do when a conditional write fails. Typically, if a condition isn't met, DynamoDB will only report an error.

However, with ReturnValuesOnConditionCheckFailure, you can choose to have DynamoDB return the item's original attributes before the write attempt failed. You have two options:

• ALL_OLD: Returns all attributes of the item.

• NONE: Returns no data (the default).

Update: Modifying Existing Items

Update Performs an UpdateItem operation to modify an existing item or create a new one if it doesn't exist. This action allows you to conditionally add, delete, or update attributes.

 private Update updateProductQuantity() {

        // products table partition key and sort key
        HashMap<String, AttributeValue> productsKey = new HashMap<>();
        productsKey.put("product_id", AttributeValue.fromS("p_123")); // partition key
        productsKey.put("name", AttributeValue.fromS("laptop")); // sort key

        // expression attribute values is fill the expected values in the query placeholder.
        Map<String, AttributeValue> productsExpressionAttributeValue = new HashMap<>();
        productsExpressionAttributeValue.put(":available_quantity", AttributeValue.fromN("11"));
        productsExpressionAttributeValue.put(":expected_product_status", AttributeValue.fromBool(true));

        return Update.builder()
                .tableName("products")
                .key(productsKey)
                // query - set quantity to 11
                .updateExpression("SET quantity = :available_quantity")
                // query - just like a where clause
                .conditionExpression("product_status = :expected_product_status")
                // provide actual values of placeholder used in the query i.e., available_quantity and  expected_product_status
                .expressionAttributeValues(productsExpressionAttributeValue)
                // An optional parameter that returns the item attributes for an UpdateItem operation that failed a condition check.
                .returnValuesOnConditionCheckFailure(ReturnValuesOnConditionCheckFailure.ALL_OLD)
                .build();
    }

updateExpression() method

The updateExpression method in the Update class of the AWS SDK for DynamoDB is used to specify how you want to modify an existing item in a table. It allows you to:

• Set new values for attributes.

• Add values to sets or numbers.

• Remove attributes or elements from sets.

• Delete elements from lists or maps

You can use placeholders in your updateExpression and then define the corresponding values using the expressionAttributeValues method. This provides a flexible way to update items based on specific conditions or requirements.

Delete: Removing Items from the Table

Delete Performs a DeleteItem operation on a specific item within a table based on its primary key.

 private Delete deleteCartEntry() {

        // cart table partition key and sort key
        HashMap<String, AttributeValue> cartKey = new HashMap<>();
        cartKey.put("cart_id", AttributeValue.fromS("ct_123")); // partition key
        cartKey.put("product_id", AttributeValue.fromS("p_123")); // sort key

        // expression attribute values is fill the expected values in the query placeholder.
        Map<String, AttributeValue> cartExpressionAttributeValue = new HashMap<>();
        cartExpressionAttributeValue.put(":expected_quantity", AttributeValue.fromN("1"));

        return Delete.builder()
                .tableName("cart")
                .key(cartKey)
                // query - just like where clause, here verifying the quantity is equal to 1
                .conditionExpression("quantity = :expected_quantity")
                // provide actual values of placeholder used in the query i.e expected_quantity which is 1
                .expressionAttributeValues(cartExpressionAttributeValue)
                // An optional parameter that returns the item attributes for an UpdateItem operation that failed a condition check.
                .returnValuesOnConditionCheckFailure(ReturnValuesOnConditionCheckFailure.ALL_OLD)
                .build();

    }

Within the AWS SDK for Java, the ConditionCheck, Put, Update, and Delete classes found in the software.amazon.awssdk.services.dynamodb.model package.(alert-success)

Executing the Transaction

Executing transactions in DynamoDB using TransactionWriteItem ensures atomic operations across multiple items.

• Create a list of TransactWriteItem objects: Each object defines a single write operation (put, update, or delete) on an item.

• Execute the transaction: Call the TransactWriteItems API, passing in the list of TransactWriteItemsRequest objects.

  public void transactionWrite() {

        // Collect Transaction Write in a list
        List<TransactWriteItem> transactionWriteItem = List.of(TransactWriteItem.builder().conditionCheck(customerExistsConditionCheck()).build(),
                TransactWriteItem.builder().put(placeOrder()).build(),
                TransactWriteItem.builder().update(updateProductQuantity()).build(),
                TransactWriteItem.builder().delete(deleteCartEntry()).build());

        // TransactWriteItemRequest
        TransactWriteItemsRequest transactWriteItemsRequest = TransactWriteItemsRequest.builder()
                .transactItems(transactionWriteItem)
                .build();

        // Run the transaction write item
        try
        {
            TransactWriteItemsResponse transactWriteItemsResponse = dynamoDbClient.transactWriteItems(transactWriteItemsRequest);
            System.out.println("Successfully completed write operation");
        } catch (Exception exception) {
            exception.printStackTrace();
        }
    }

A TransactionCanceledException exception may occur due to any of the following reasons
• When a condition in one of the condition expressions is not met.
• When a transaction validation error occurs because more than one action in the same TransactWriteItems operation targets the same item.
• When a TransactWriteItems request conflicts with an ongoing TransactWriteItems operation on one or more items in the TransactWriteItems request.(alert-error)

TransactGetItems API

TransactGetItems efficiently fetches data from multiple DynamoDB tables by combining up to 100 individual Get operations into a single synchronous request.

This allows for retrieving data from a maximum of 100 items across different tables within the same AWS account and Region, provided the total data size doesn't exceed 4 MB.

The Get actions are performed atomically so that either all of them succeed or all of them fail:

Get — Initiates a GetItem operation to retrieve a set of attributes for the item with the given primary key. If no matching item is found, Get does not return any data.

To illustrate the TransactGetItems operation, I will retrieve data from both the customer and orders tables in the following example.

  public void transactionRead() {
        // customer's partition key and sort key
        HashMap<String, AttributeValue> customerKey = new HashMap<>();
        customerKey.put("customer_id", AttributeValue.fromS("c_123")); // partition Key
        customerKey.put("email", AttributeValue.fromS("dummy@email.com")); // sort key

        // order's partition key and sort key
        Map<String, AttributeValue> orderKey = new HashMap<>();
        orderKey.put("order_id", AttributeValue.fromS("od_123")); // partition Key
        orderKey.put("product_id", AttributeValue.fromS("p_123")); // sort key

        // Fetch all the customer whose partition key and sort key match
        Get customersGet = Get.builder()
                .tableName("customer")
                .key(customerKey)
                .build();

        // Fetch all the orders whose partition key and sort key match
        Get ordersGet = Get.builder()
                .tableName("orders")
                .key(orderKey)
                .build();

        // Collect all the read operation in a list of TransactGetItem
        List<TransactGetItem> transactGetItemList = List.of(TransactGetItem.builder().get(customersGet).build(),
                TransactGetItem.builder().get(ordersGet).build());

        // TransactGetItemRequest
        TransactGetItemsRequest transactGetItemsRequest = TransactGetItemsRequest.builder()
                .transactItems(transactGetItemList)
                .returnConsumedCapacity(ReturnConsumedCapacity.TOTAL)
                .build();

        // Run the transactGetItem operation
        try {
            TransactGetItemsResponse transactGetItemsResponse = dynamoDbClient.transactGetItems(transactGetItemsRequest);
            List<ItemResponse> responses = transactGetItemsResponse.responses();
            if (Objects.nonNull(responses) && !responses.isEmpty()) {
                responses.stream()
                        .flatMap(response -> response.item().entrySet().stream())
                        .forEach(entry -> {
                            System.out.println("attribute: " + entry.getKey() + " value: " + entry.getValue());
                        } );
            }
        } catch (Exception exception) {
            exception.printStackTrace();
        }
    }

Read transactions are unsuccessful in the following cases:
• When a TransactGetItems request conflicts with an ongoing TransactWriteItems operation on one or more items in the TransactGetItems request. In this case, the request fails with a TransactionCanceledException.(alert-error)

Conclusion: Leveraging TransactWriteItems and TransactGetItems for Robust DynamoDB Transactions

This article has demonstrated the effective use of TransactWriteItems and TransactGetItems to execute complex transactions within DynamoDB.

By understanding the core concepts, best practices, and potential limitations, developers can confidently implement atomic operations to maintain data integrity and consistency in their applications.

Enhance your DynamoDB table's performance and lower operational costs by implementing (getButton) #text=(DynamoDB's Time to Live for Automatic Data Deletion) #icon=(link) #color=(#35a576) feature.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

How to Enable Time-to-Live (TTL) for Automatic Data Deletion in Amazon DynamoDB Table

2024-11-24T20:11:00.000+05:30

By leveraging DynamoDB's Time-to-Live (TTL) feature, you can implement a cost-effective strategy for automated data retention. By setting an expiration timestamp for each item, DynamoDB will automatically delete it once the specified time has elapsed.

This process is transparent to the application and does not require additional write capacity units.

Enabling Time to Live (TTL) on a DynamoDB table is easy. You can do this using the AWS Management Console, AWS CLI, or the AWS API. The AWS Console offers the most user-friendly experience.

~~toc~~

Why Time to Live (TTL) is Essential for DynamoDB Performance

While you can manually delete items from your DynamoDB table using various methods, configuring TTL offers significant advantages in terms of performance and cost-effectiveness.

Manual deletion operations, regardless of the method used, consume write throughput capacity. This can be particularly expensive when dealing with large numbers of items.

TTL, however, enables automatic item deletion without consuming any write throughput, leading to substantial savings.

Please note that DynamoDB can take up to 48 hours or more to fully process and delete items marked for expiration.(alert-warning)

Enable Time to Live (TTL) using the AWS CLI

The most straightforward approach to enable Time to Live (TTL) on a DynamoDB table is through the AWS CLI. I'll apply this to my session_data table, utilizing the expires_at attribute to automatically expire outdated records.

 aws dynamodb update-time-to-live \ 
 --table-name session_data \ 
 --time-to-live-specification "Enabled=true, AttributeName=expires_at"

The following output will be displayed after executing the AWS CLI command above.

 {
    "TimeToLiveSpecification": {
        "Enabled": true,
        "AttributeName": "expires_at"
    }
  }

If you missed the previous output, you can alternatively describe the TTL configuration for a DynamoDB table to retrieve the information.

 aws dynamodb describe-time-to-live --table-name session_data

Enable Time to Live (TTL) using the AWS console

To leverage DynamoDB's automatic item deletion feature, you need to add a timestamp attribute of type Number to your table.

The timestamp value should be in Unix epoch time format, which represents the number of seconds elapsed since January 1, 1970 UTC. DynamoDB exclusively supports this format for automatic TTL.

I'll leverage an existing DynamoDB table named session_data for this exercise. The expire_at attribute within this table will be used to set up Time to Live (TTL) and automate item deletion.

STEP 1: From the list of available DynamoDB tables, select the table you want to configure for Time to Live (TTL). In my example, I've chosen the session_data table.

STEP 2: Select "Update Setting" from the "Actions" dropdown menu, as shown in the screenshot.

STEP 3: Once the DynamoDB table configuration page opens, you'll see the "Overview" section selected by default. Click the "Additional settings" button located in the top-right corner (highlighted in red).

STEP 4: Locate the "Time to Live (TTL)" section by scrolling down the page. By default, TTL is disabled. To activate it, click the "Turn On" button.

STEP 5: In the TTL settings, specify the Number type attribute (e.g., expire_at) that will be used to determine item expiration based on Unix epoch timestamps.

STEP 6: Optionally, To test the TTL configuration before activating it, click the "Run Preview" button within the "Preview" section. This will simulate item deletion based on the specified TTL settings. Once you're satisfied with the preview, click "Turn On TTL" to enable the configuration.

Once you've activated TTL, the table configuration page will reflect the updated status: "TTL: On".

Conclusion

Time to Live (TTL) is a powerful feature in DynamoDB that can optimize performance and minimize costs. By automatically deleting expired items, TTL reduces storage overhead and improves query performance.

This tutorial has guided you through the process of enabling TTL on your DynamoDB table using the AWS Management Console.

Keep learning and keep growing.

How to Read Amazon S3 Files Directly in Java/Spring Boot Without Downloading

2024-10-26T20:57:00.000+05:30

Amazon Simple Storage Service (S3) offers a reliable and scalable solution for object storage in the cloud.

In today's cloud-native world, developers frequently leverage cloud storage services like Amazon S3 to store and manage files efficiently. However, traditional approaches involving downloading files locally can introduce performance bottlenecks.

In this tutorial, we'll explore how to streamline your workflow and boost application speed by directly reading files from S3, bypassing the need for local file transfers.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Dependencies

To get started with this tutorial, you'll need to add the following dependencies to your Spring Boot project's. This dependency provides the necessary components for interacting with AWS services, including S3.

Gradle

ext {
    springCloudAwsVersion = '3.0.0'
}

dependencies {
    // spring cloud aws BOM(Bill of materials)
    implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
    // AWS launched a high level file transfer utility, called Transfer Manager and a CRT based S3 client.
    // The starter automatically configures and registers a software.amazon.awssdk.transfer.s3.S3TransferManager bean if the following dependency is added to the project:
    implementation 'software.amazon.awssdk:s3-transfer-manager'
    //Transfer Manager works the best with CRT S3 Client. To auto-configure CRT based S3AsyncClient add following dependency to your project:
    implementation 'software.amazon.awssdk.crt:aws-crt'
}

Maven

<dependencies>
    <dependency>
        <groupId>io.awspring.cloud</groupId>
        <artifactId>spring-cloud-aws-starter-s3</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3-transfer-manager</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws-dependencies</artifactId>
            <version>{project-version}</version>
            <type>pom</type>
            <scope>import</scope>
       </dependency>
    </dependencies>
</dependencyManagement>

Configuration and Credentials

Spring Cloud AWS offers auto-configuration for S3Client, S3TransferManager, and S3Template, making setup a breeze. Here's what you need to add to your application.properties file:

 # s3 Configuration
 spring.cloud.aws.credentials.access-key=
 spring.cloud.aws.credentials.secret-key=
 # Configures endpoint used by S3Client, I'm woorking in the Asia Pacific (Mumbai) hence, I've configure ap-south1 region and endpoint
 spring.cloud.aws.s3.endpoint=https://s3.ap-south-1.amazonaws.com
 spring.cloud.aws.region.static=ap-south-1

IAM Permissions for S3 Read Access

To grant your IAM user or role permission to read files from S3, you must attach this policy to it.

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "readAccess",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::*/*"
        }
    ]
 }

Reading S3 Objects Directly into Java Streams

S3Client is a Java interface for interacting with Amazon S3. We'll use S3Client to directly read the S3 object into an input stream. Then, we'll use a BufferedReader to process the file line by line.

 
 import software.amazon.awssdk.core.ResponseInputStream;
 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.GetObjectRequest;
 import software.amazon.awssdk.services.s3.model.GetObjectResponse;

 import java.io.BufferedReader;
 import java.io.InputStream;
 import java.io.InputStreamReader;
 import java.util.Arrays;
 import java.util.Objects;

 public void readFileUsingS3Client(String bucketName, String key) {
    try {
        GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();

        ResponseInputStream<GetObjectResponse> responseResponseInputStream = s3Client.getObject(getObjectRequest);
        BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(responseResponseInputStream));

        String line = "";
        while (Objects.nonNull(line = bufferedReader.readLine())) {
            // Since the file format is CSV, we'll split each line by commas (,) to access individual columns.
            String[] split = line.split(",");
            System.out.println("line: " + Arrays.toString(split));
        }
    } catch (Exception exception) {
        exception.printStackTrace();
    }
 }

The getObject() method of S3Client allows us to read an S3 object directly into a Java input stream. It takes a GetObjectRequest object as an argument, specifying the bucket name and object key.

Conclussion

Reading a file directly from Amazon S3 can not only boost application processing time but also optimize resource utilization, reducing latency and optimizing memory usage by eliminating the need to download files locally

To enable public access to your S3 objects without exposing your AWS credentials, explore (getButton) #text=(how to generate presigned URLs) #icon=(link) #color=(#35a576) in our comprehensive guide.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

Spring Cloud AWS S3 File Download Tutorial: Step-by-Step Guide

2024-10-26T20:55:00.000+05:30

Amazon S3 is a scalable, object-based storage service. Today's enterprise applications often leverage serverless storage solutions like S3 to store and retrieve files efficiently.

This tutorial will demonstrate how to download files or directories from an Amazon S3 bucket using Spring Cloud AWS.

Spring Cloud AWS provides a convenient way to integrate S3 functionality into Spring Boot applications.

To download files from Amazon S3, you'll first need to (getButton) #text=(upload files to your S3 bucket) #icon=(link) #color=(#35a576) Then, follow the steps below

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Dependencies

Gradle

ext {
    springCloudAwsVersion = '3.0.0'
}

dependencies {
    // spring cloud aws BOM(Bill of materials)
    implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
    // AWS launched a high level file transfer utility, called Transfer Manager and a CRT based S3 client.
    // The starter automatically configures and registers a software.amazon.awssdk.transfer.s3.S3TransferManager bean if the following dependency is added to the project:
    implementation 'software.amazon.awssdk:s3-transfer-manager'
    //Transfer Manager works the best with CRT S3 Client. To auto-configure CRT based S3AsyncClient add following dependency to your project:
    implementation 'software.amazon.awssdk.crt:aws-crt'
}

Maven

<dependencies>
    <dependency>
        <groupId>io.awspring.cloud</groupId>
        <artifactId>spring-cloud-aws-starter-s3</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3-transfer-manager</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws-dependencies</artifactId>
            <version>{project-version}</version>
            <type>pom</type>
            <scope>import</scope>
       </dependency>
    </dependencies>
</dependencyManagement>

Configuration and Credentials

Before initiating the object download process, let's configure our Spring Boot application to interact with AWS S3. Spring Cloud AWS offers auto-configuration for S3Client, S3TransferManager, and S3Template, making setup a breeze.

Here's what you need to add to your application.properties file:

 # s3 Configuration
 spring.cloud.aws.credentials.access-key=
 spring.cloud.aws.credentials.secret-key=
 # Configures endpoint used by S3Client, I'm woorking in the Asia Pacific (Mumbai) hence, I've configure ap-south1 region and endpoint
 spring.cloud.aws.s3.endpoint=https://s3.ap-south-1.amazonaws.com
 spring.cloud.aws.region.static=ap-south-1

IAM Permissions: Downloading S3 Objects

Each AWS service requires specific permissions to perform operations. These permissions can be granted to IAM users through policies, AWS service roles, or other mechanisms.

For this tutorial, we'll use an IAM user to download an object from S3. To enable this, we've attached the necessary policy to the IAM user. Alternatively, you can use an IAM role

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DownloadFilePolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucketVersions",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::*/*",
                "arn:aws:s3:::learnjavaskills"
            ]
        }
    ]
 }

Downloading Files from AWS S3 with S3Client

Let's explore our first approach to downloading an object from Amazon S3. We'll utilize the S3Client interface, a dedicated service client for interacting with the S3 service.

The S3Client interface is a fundamental component of the AWS SDK for Java. It provides a high-level API for interacting with Amazon Simple Storage Service (S3). This interface allows developers to perform various operations on S3 buckets and objects

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.GetObjectRequest;
 import java.nio.file.Path;

 public void downloadFileUsingS3Client(String bucketName, String key, String downloadDestinationPath) {
    try {
        GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();
        s3Client.getObject(getObjectRequest, Path.of(downloadDestinationPath));
    } catch (Exception exception) {
        exception.printStackTrace();
    }
 }

The getObject() method of S3Client is the primary mechanism for downloading objects from an S3 bucket. It requires two parameters: a GetObjectRequest object specifying the bucket name and object key, and a local file path where the downloaded object will be saved.

Downloading Files from AWS S3 with S3TransferManager

S3TransferManager is a powerful utility within the AWS SDK for Java that simplifies the process of transferring files to and from Amazon S3.

It offers enhanced performance and reliability by leveraging Amazon S3's multipart upload and byte-range fetch capabilities.

 import software.amazon.awssdk.services.s3.model.GetObjectRequest;
 import software.amazon.awssdk.transfer.s3.S3TransferManager;
 import software.amazon.awssdk.transfer.s3.model.DownloadFileRequest;
 import software.amazon.awssdk.transfer.s3.model.FileDownload;
 import software.amazon.awssdk.transfer.s3.model.CompletedFileDownload;
 import java.nio.file.Path;

 public void downloadFileUsingS3TransferManager(String bucketName, String key, String downloadDestinationPath) {
    try {
        GetObjectRequest getObjectRequest = GetObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();

        DownloadFileRequest downloadFileRequest = DownloadFileRequest.builder()
                .destination(Path.of(downloadDestinationPath))
                .getObjectRequest(getObjectRequest)
                .build();
        FileDownload fileDownload = s3TransferManager.downloadFile(downloadFileRequest);
        CompletedFileDownload completedFileDownload = fileDownload.completionFuture()
                .join();
        String contentType = completedFileDownload.response()
                .contentType();
        System.out.println("Download completed, content type is : " + contentType);
    } catch (Exception exception) {
        exception.printStackTrace();
    }
 }

The downloadFile() method of S3TransferManager facilitates downloading S3 objects. It requires a DownloadFileRequest object as an argument.

This request specifies the object's details, including the local destination path and an GetObjectRequest instance containing the bucket name and key, as discussed in our previous example.

Understanding completionFuture()

The completionFuture method in the FileDownload interface of the AWS SDK for Java provides a way to asynchronously track the progress and completion of an S3 file download operation.

It returns a CompletableFuture<Void> object, which represents the future result of the download operation. Key points about completionFuture:

• Asynchronous Execution

• Progress Tracking

• Error Handling

• Completion Notification

Downloading Directories from AWS S3 with S3TransferManager

Similarly, we can efficiently download entire directories or buckets from Amazon S3 using S3TransferManager. To accomplish this, we employ the downloadDirectory() method instead of downloadFile() method.

 import software.amazon.awssdk.transfer.s3.S3TransferManager;
 import software.amazon.awssdk.transfer.s3.model.DownloadDirectoryRequest;
 import software.amazon.awssdk.transfer.s3.model.DirectoryDownload;
 import software.amazon.awssdk.transfer.s3.model.CompletedDirectoryDownload;
 import java.nio.file.Path;

 public void downloadDirectoryUsingS3TransferManager(String bucketName, String downloadDestinationPath) {
    try {
        DownloadDirectoryRequest downloadDirectoryRequest = DownloadDirectoryRequest.builder()
                .bucket(bucketName)
                .destination(Path.of(downloadDestinationPath))
                .build();
        DirectoryDownload directoryDownload = s3TransferManager.downloadDirectory(downloadDirectoryRequest);
        CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture()
                .join();
        completedDirectoryDownload.failedTransfers()
                .forEach(failedFileDownload -> System.out.println("fail to download : " + failedFileDownload.toString()));
    } catch (Exception exception) {
        exception.printStackTrace();
    }
 }

Downloading Files with S3Template

The S3Template class in the Spring Cloud AWS library provides a high-level abstraction for interacting with Amazon S3. It simplifies common S3 operations, making it easier to integrate S3 functionality into your Spring Boot applications.

The S3Template provides a convenient download() method for retrieving objects from Amazon S3. This method requires two arguments: the bucket name and the object key, which uniquely identify the desired object within the S3 storage.

 import io.awspring.cloud.s3.S3Resource;
 import io.awspring.cloud.s3.S3Template;
 import java.io.InputStream;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.StandardCopyOption;

 public void downloadFileUsingS3Template(String bucketName, String key, String downloadDestinationPath) {
    try {
        S3Resource s3Resource = s3Template.download(bucketName, key);
        InputStream inputStream = s3Resource.getInputStream();
        Files.copy(inputStream, Path.of(downloadDestinationPath), StandardCopyOption.REPLACE_EXISTING);
    } catch (Exception exception) {
        exception.printStackTrace();
    }
 }

Conclussion

For Spring Boot applications requiring single file downloads, S3Template offers an efficient and convenient approach.

In legacy Java applications without Spring Cloud AWS integration, consider using S3Client for object downloads.

S3TransferManager provides the most efficient and feature-rich option for object downloads, offering benefits like progress tracking, error handling, and asynchronous communication, making it well-suited for multithreaded environments.

Learn how to efficiently (getButton) #text=(read files directly into a Java input stream) #icon=(link) #color=(#35a576) without the need for downloading

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

Spring Cloud AWS S3 File Upload Tutorial: Step-by-Step Guide

2024-10-12T20:11:00.000+05:30

In today's digital age, storing and managing large amounts of data is crucial for many applications. Amazon Simple Storage Service (S3) offers a reliable and scalable solution for object storage in the cloud.

Spring Cloud AWS provides a convenient way to integrate S3 functionality into Spring Boot applications.

This article will guide you through the process of uploading files to Amazon S3 using Spring Cloud AWS, covering essential steps and configurations. By the end, you'll have a solid understanding of how to leverage S3 storage for your Spring Boot applications.

To get started with uploading files to Amazon S3, make sure you have a bucket ready. Refer to my previous blog posts for instructions on (getButton) #text=(creating, deleting, and managing buckets) #icon=(link) #color=(#35a576)

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Dependencies

Gradle

ext {
    springCloudAwsVersion = '3.0.0'
}

dependencies {
    // spring cloud aws BOM(Bill of materials)
    implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
    // AWS launched a high level file transfer utility, called Transfer Manager and a CRT based S3 client.
    // The starter automatically configures and registers a software.amazon.awssdk.transfer.s3.S3TransferManager bean if the following dependency is added to the project:
    implementation 'software.amazon.awssdk:s3-transfer-manager'
    //Transfer Manager works the best with CRT S3 Client. To auto-configure CRT based S3AsyncClient add following dependency to your project:
    implementation 'software.amazon.awssdk.crt:aws-crt'
}

Maven

<dependencies>
    <dependency>
        <groupId>io.awspring.cloud</groupId>
        <artifactId>spring-cloud-aws-starter-s3</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3-transfer-manager</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws-dependencies</artifactId>
            <version>{project-version}</version>
            <type>pom</type>
            <scope>import</scope>
       </dependency>
    </dependencies>
</dependencyManagement>

Configuration and Credentials

Before we get started with uploading files, let's configure our Spring Boot application to interact with AWS S3. Spring Cloud AWS offers auto-configuration for S3Client, S3TransferManager, and S3Template, making setup a breeze.

Here's what you need to add to your application.properties file:

 # s3 Configuration
 spring.cloud.aws.credentials.access-key=
 spring.cloud.aws.credentials.secret-key=
 # Configures endpoint used by S3Client, I'm woorking in the Asia Pacific (Mumbai) hence, I've configure ap-south1 region and endpoint
 spring.cloud.aws.s3.endpoint=https://s3.ap-south-1.amazonaws.com
 spring.cloud.aws.region.static=ap-south-1

IAM Permissions for Bucket Management

In this in-depth tutorial, we'll explore various features of AWS S3, including tagging, metadata, and file uploading. We'll delve into multiple approaches for uploading files.

To seamlessly upload files with tags and metadata, your Amazon Simple Storage Service (S3) bucket will require the following IAM permissions.

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "uploadFile",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObjectTagging"
            ],
            "Resource": "arn:aws:s3:::*/*"
        }
    ]
 }

AWS S3 File Upload with S3Client

One of the most straightforward ways to upload a file to Amazon Simple Storage Service (S3) is by using the S3Client interface. S3Client is a service client designed for interacting with Amazon S3.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.PutObjectRequest;
 import software.amazon.awssdk.core.sync.RequestBody;
 import java.nio.file.Paths;

 public void uploadFileUsingS3Client(String bucketName, String key, String filePath) {
    try {
        PutObjectRequest putObjectRequest = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();
        s3Client.putObject(putObjectRequest, RequestBody.fromFile(Paths.get(filePath)));
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
    }
 }

The putObject() method of S3Client is your gateway to uploading files to your Amazon S3 bucket. It takes two arguments: a PutObjectRequest object and a RequestBody, which specifies the exact file you want to upload.

• PutObjectRequest is a class (located in the com.amazonaws.services.s3.model package) that allows you to configure the upload process.

• To create a valid PutObjectRequest, specifying the target bucket name and the key (unique identifier) for the file is essential.

Running Unit Tests for File Uploading

 @Test
 void uploadFileUsingS3Client()
 {
    uploadObjects.uploadFileUsingS3Client(bucketName, "robot.gif", filePath);
 }

File Upload Successful to Amazon S3

Efficient File Uploads and Tagging with S3Client

Tagging in Amazon S3 is a powerful feature that allows you to attach metadata to your objects. These metadata tags are key-value pairs that provide additional information about your objects, making them easier to categorize, search, and manage.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.PutObjectRequest;
 import software.amazon.awssdk.services.s3.model.Tag;
 import software.amazon.awssdk.services.s3.model.Tagging;
 import software.amazon.awssdk.core.sync.RequestBody;
 import java.nio.file.Paths;
 import java.util.List;

 public void uploadFileWithTag(String bucketName, String key, String filePath) {
    try {
        Tag tag1 = Tag.builder().
                key("tag 1")
                .value("tag 1 value")
                .build();
        Tag tag2 = Tag.builder()
                .key("tag 2")
                .value("tag 2 value")
                .build();

        List<Tag> tagList = List.of(tag1, tag2);
        Tagging tagging = Tagging.builder()
                .tagSet(tagList)
                .build();

        PutObjectRequest putObjectRequest = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .tagging(tagging)
                .build();
        s3Client.putObject(putObjectRequest, RequestBody.fromFile(Paths.get(filePath)));
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
    }
 }

To tag an object in Amazon S3, simply use the Tagging class to add your desired key-value pairs while uploading the file. The Tagging class has a tagSet() method that accepts a list of Tag objects.

To tag an object while uploading it to Amazon S3, use the tagging() method of the PutObjectRequest class to provide the desired tagging information.

Testing File Uploads with Tags

 @Test
 void uploadFileWithTag()
 {
    uploadObjects.uploadFileWithTag(bucketName, "robot-with-tag.gif", filePath);
 }

Verifying Tagged File Uploads

Efficiently Updating Tag Information for S3 Files

You may need to update or modify the tag information associated with an Amazon S3 object. To do this, you must first retrieve the existing tags from the S3 bucket.

You can then update or create new tags as needed. Once you've configured the desired tag information, you can apply these changes to the S3 object by using the PutObjectTaggingRequest class.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.PutObjectRequest;
 import software.amazon.awssdk.services.s3.model.GetObjectTaggingRequest;
 import software.amazon.awssdk.services.s3.model.GetObjectTaggingResponse;
 import software.amazon.awssdk.services.s3.model.PutObjectTaggingRequest;
 import software.amazon.awssdk.services.s3.model.Tag;
 import software.amazon.awssdk.services.s3.model.Tagging;
 import java.util.List;

 public void updateObjectTag(String bucketName, String key) {
    try {
        GetObjectTaggingRequest getObjectTaggingRequest = GetObjectTaggingRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();
        GetObjectTaggingResponse objectTagging = s3Client.getObjectTagging(getObjectTaggingRequest);

        // print existing tags
        objectTagging.tagSet()
                .stream()
                .forEach(tag -> System.out.println("tag key : " + tag.key() + " tag value : " + tag.value()));

        // new tags add
        Tag tag3 = Tag.builder()
                    .key("tag 3")
                    .value("tag 3 value")
                    .build();
        Tag tag4 = Tag.builder()
                    .key("tag 4")
                    .value("tag 4 value")
                    .build();

        List<Tag> tagList = List.of(tag3, tag4);
        Tagging updatedTags = Tagging.builder()
                .tagSet(tagList)
                .build();

        PutObjectTaggingRequest putObjectTaggingRequest = PutObjectTaggingRequest.builder()
                .bucket(bucketName)
                .key(key)
                .tagging(updatedTags)
                .build();

        // update tags
        s3Client.putObjectTagging(putObjectTaggingRequest);
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
    }
 }

Testing Tag Updates in Amazon S3

 @Test
 void updateObjectTag()
 {
    uploadObjects.updateObjectTag(bucketName, "robot-with-tag.gif");
 }

Tag Update Successful

AWS S3 File Upload with S3Client: Adding Metadata to Uploading Files

Metadata is a set of data that describes an object stored in Amazon S3. It provides additional information about the object, such as its creation date, size, content type, and custom user-defined attributes.

This metadata can be used to organize, search, and manage your objects more effectively.

Key types of metadata in S3:

• System-defined metadata: This metadata is automatically created and managed by S3 and includes information like object size, creation date, and storage class.

• User-defined metadata: This metadata is added by the user and can contain any key-value pairs. It's a flexible way to store custom information about your objects.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.PutObjectRequest;
 import software.amazon.awssdk.core.sync.RequestBody;
 import java.nio.file.Paths;
 import java.util.Map;

 public void uploadFileWithMetaData(String bucketName, String key, String filePath) {
    try {
        Map<String, String> metadata = new HashMap<>();
        metadata.put("author", "learnjavaskills.in");
        metadata.put("file-type", "txt");
        metadata.put("version", "1.0");

        PutObjectRequest putObjectRequest = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .metadata(metadata)
                .build();
        s3Client.putObject(putObjectRequest, RequestBody.fromFile(Paths.get(filePath)));
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
    }
 }

Testing File Uploads with Metadata in Amazon S3

 @Test
 void uploadFileWithMetaData()
 {
    uploadObjects.uploadFileWithMetaData(bucketName, "robot-with-meta-data.gif", filePath);
 }

File Upload Successful with Metadata

AWS S3 File Upload with S3TransferManager

The S3TransferManager class in the AWS SDK for Java provides a high-level API for managing asynchronous data transfers to and from Amazon S3.

It handles the complexities of multi-part uploads, retries, and error handling, making it a convenient choice for large file transfers or bulk data operations.

 import software.amazon.awssdk.transfer.s3.S3TransferManager;
 import software.amazon.awssdk.transfer.s3.model.UploadFileRequest;
 import software.amazon.awssdk.transfer.s3.model.FileUpload;
 import software.amazon.awssdk.transfer.s3.model.CompletedFileUpload;
 import java.nio.file.Paths;
 import java.util.OptionalLong;

 public void uploadFileWithTransferManager(String bucketName, String key, String filePath) {
    try
    {
        UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                .putObjectRequest(builder -> builder.bucket(bucketName)
                        .key(key))
                .source(Paths.get(filePath))
                .build();
        FileUpload fileUpload = s3transferManager.uploadFile(uploadFileRequest);
        // check progress of file upload
        while (true)
        {
            OptionalLong optionalLong = fileUpload.progress()
                    .snapshot()
                    .remainingBytes();
            if (optionalLong.isPresent())
            {
                System.out.println("remaining byte transfer : " + optionalLong.getAsLong());
                if (optionalLong.getAsLong() == 0)
                    break;
            }
        }

        CompletedFileUpload completedFileUpload = fileUpload.completionFuture().join();
        boolean successful = completedFileUpload.response().sdkHttpResponse().isSuccessful();
        System.out.println("is Successful : " + successful);
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
    }
 }

AWS S3 File Upload with S3Template

A straightforward way to upload a file to Amazon Simple Storage Service (S3) is by using the upload() method provided by S3Template.

The upload() method requires three essential arguments: the bucket name, the object key, and the file's input stream.

 import io.awspring.cloud.s3.S3Template;
 import io.awspring.cloud.s3.S3Resource;
 import java.io.FileInputStream;
 import java.io.InputStream;
 import java.net.URL;

 public void uploadFileUsingS3Template(String bucketName, String key, String filePath) {
    try {
        InputStream inputStream = new FileInputStream(filePath);
        S3Resource s3Resource = s3Template.upload(bucketName, key, inputStream);
        URL url = s3Resource.getURL();
        System.out.println("File uploaded successfully at " + url.toString());
    } catch (S3Exception | IOException exception) {
        exception.printStackTrace();
    }
 }

AWS S3 File Upload with S3Template: Adding Metadata to Uploaded Files

S3Template also allows you to add custom metadata to an Amazon S3 object. An overloaded version of the upload() method accepts metadata as an argument, in addition to the bucket name, object key, and input stream of the file.

 import io.awspring.cloud.s3.S3Template;
 import io.awspring.cloud.s3.S3Resource;
 import java.io.FileInputStream;
 import java.io.InputStream;
 import io.awspring.cloud.s3.ObjectMetadata;
 import java.net.URL;

 public void uploadFileUsingS3TemplateWithMetadata(String bucketName, String key, String filePath) {
    try {
        InputStream inputStream = new FileInputStream(filePath);
        ObjectMetadata objectMetadata = ObjectMetadata.builder()
                .metadata("author", "learnjavaskills.in")
                .metadata("version", "1.0")
                .contentType("text/plain")
                .build();
        S3Resource s3Resource = s3Template.upload(bucketName, key, inputStream, objectMetadata);
        URL url = s3Resource.getURL();
        System.out.println("File uploaded successfully at " + url.toString());
    }catch (S3Exception | IOException exception) {
        exception.printStackTrace();
    }
 }

Testing File Uploads with Metadata

 @Test
 void uploadFileUsingS3TemplateWithMetadata()
 {
    uploadObjects.uploadFileUsingS3TemplateWithMetadata(bucketName, "robot-s3-template-metadata.gif", filePath);
 }

Verifying Amazon S3 Objects with Metadata

Conclussion

For Spring Boot projects, S3Template simplifies file uploads with its convenient upload() method.

S3TransferManager offers significant advantages over S3Client, including multi-part uploads, asynchronous transfers, robust error handling, and progress tracking.

While S3Client is suitable for small file uploads, S3TransferManager excels in handling larger files and complex transfer scenarios.

Now that you've successfully uploaded your files to Amazon S3, let's explore how to (getButton) #text=(download files from S3 bucket) #icon=(link) #color=(#35a576) when needed.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

Spring Cloud AWS S3 Bucket Operations (Create, List, Delete)

2024-10-12T17:12:00.000+05:30

In this blog post, we'll delve into the world of Spring Boot and AWS S3 bucket management. We'll explore how to seamlessly integrate AWS S3 into your Spring Boot applications using Spring Cloud AWS, a powerful framework that simplifies the process of interacting with AWS services.

By the end of this tutorial, you'll have a solid understanding of how to list, create, and delete S3 buckets using Spring Cloud AWS, enabling you to efficiently manage your cloud storage needs.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Dependencies

Gradle

ext {
    springCloudAwsVersion = '3.0.0'
}

dependencies {
    // spring cloud aws BOM(Bill of materials)
    implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
    // AWS launched a high level file transfer utility, called Transfer Manager and a CRT based S3 client.
    // The starter automatically configures and registers a software.amazon.awssdk.transfer.s3.S3TransferManager bean if the following dependency is added to the project:
    implementation 'software.amazon.awssdk:s3-transfer-manager'
    //Transfer Manager works the best with CRT S3 Client. To auto-configure CRT based S3AsyncClient add following dependency to your project:
    implementation 'software.amazon.awssdk.crt:aws-crt'
}

Maven

<dependencies>
    <dependency>
        <groupId>io.awspring.cloud</groupId>
        <artifactId>spring-cloud-aws-starter-s3</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3-transfer-manager</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws-dependencies</artifactId>
            <version>{project-version}</version>
            <type>pom</type>
            <scope>import</scope>
       </dependency>
    </dependencies>
</dependencyManagement>

Configuration and Credentials

Before diving into bucket operations, let's configure our Spring Boot application to interact with AWS S3.

Spring Cloud AWS offers auto-configuration for S3Client, S3TransferManager, and S3Template, making setup a breeze. Here's what you need to add to your application.properties file:

 # s3 Configuration
 spring.cloud.aws.credentials.access-key=
 spring.cloud.aws.credentials.secret-key=
 # Configures endpoint used by S3Client, I'm woorking in the Asia Pacific (Mumbai) hence, I've configure ap-south1 region and endpoint
 spring.cloud.aws.s3.endpoint=https://s3.ap-south-1.amazonaws.com
 spring.cloud.aws.region.static=ap-south-1

IAM Permissions for Bucket Management

You must apply this policy to your IAM user or role to enable listing, creation, and deletion of buckets.

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListCreateDeleteBucket",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteBucketWebsite",
                "s3:CreateBucket",
                "s3:ListBucket",
                "s3:DeleteBucket",
                "s3:GetObject",
                "s3:DeleteObjectVersion",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::*"
        }
    ]
 }

Creating an S3 Bucket with S3Client

S3Client is a Java interface that interacts with Amazon Simple Storage Service (S3). An S3 bucket is like a virtual folder where you can store any type of file. Buckets have unique names globally. Let's create your first bucket on Amazon S3.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.*;

 public void createBucket(String bucketName) {
    try {
        CreateBucketRequest createBucketRequest = CreateBucketRequest.builder()
                .bucket(bucketName)
                .build();
        CreateBucketResponse createBucketResponse = s3Client.createBucket(createBucketRequest);
        boolean successful = createBucketResponse.sdkHttpResponse()
                .isSuccessful();
        if (successful) {
            String location = createBucketResponse.location();
            System.out.println("Bucket created successfully at : " + location);
        } else System.out.println("Unable to create bucket");
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
    }
 }

Listing All Buckets Using S3Client

Using the S3Client, you can retrieve a list of all buckets in your configured region. The listBuckets() method provides this functionality.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.*;

 public void listBucketObjects() {
    try {
        ListBucketsResponse listBucketsResponse = s3Client.listBuckets();
        listBucketsResponse.buckets()
                .stream()
                .forEach(bucket -> System.out.println("bucket : " + bucket.name()));
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
    }
 }

Deleting Empty Buckets with S3Client

To delete a bucket on Amazon S3, use the deleteBucket() method of the S3Client. However, the bucket must be empty before deletion; otherwise, the AWS SDK will throw an exception

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.*;

 public boolean deleteEmptyBucket(String bucketName) {
    try {
        DeleteBucketRequest deleteBucketRequest = DeleteBucketRequest.builder()
                .bucket(bucketName)
                .build();
        DeleteBucketResponse deleteBucketResponse = s3Client.deleteBucket(deleteBucketRequest);
        boolean successful = deleteBucketResponse.sdkHttpResponse()
                .isSuccessful();
        System.out.println("is bucket deleted : " + successful);
        return successful;
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
        return false;
    }
 }

When deleting an S3 bucket using the AWS SDK for Java, a software.amazon.awssdk.services.s3.model.S3Exception is thrown if the bucket is not empty. This exception signifies a validation error and prevents accidental deletion of buckets with content.(alert-error)

Deleting Unversioned, Non-Empty S3 Buckets with S3Client

If you have a non-empty bucket without versioning enabled, you'll need to delete its contents before deleting the bucket itself. To do this, list all the objects in the bucket and then delete them individually.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.*;

 public boolean deleteUnversionedBucket(String bucketName) {
    try {
        // Before you can delete an Amazon S3 bucket, you must ensure that the bucket is empty or an error will result.
        // If you have a versioned bucket, you must also delete any versioned objects associated with the bucket.
        ListObjectsRequest listObjectsRequest = ListObjectsRequest.builder()
                .bucket(bucketName)
                .build();
        ListObjectsResponse listObjectsResponse = s3Client.listObjects(listObjectsRequest);
        listObjectsResponse.contents()
                .stream()
                // create a DeleteObjectRequest
                .map(s3Object -> DeleteObjectRequest.builder()
                        .bucket(bucketName)
                        .key(s3Object.key())
                        .build())
                // delete objects
                .forEach(deleteObjectRequest -> s3Client.deleteObject(deleteObjectRequest));

        // once all the objects in a bucket deleted successfully, we can delete empty bucket.
        return deleteEmptyBucket(bucketName);
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
        return false;
    }
 }

Deleting a Versioned Non-Empty S3 Bucket with S3Client

If versioning is enabled for your bucket, you must delete all versioned objects individually before deleting the bucket itself. Attempting to delete a versioned bucket directly will result in an S3Exception.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.*;

 public boolean deleteVersionedBucket(String bucketName) {
    try {
        // Before you can delete an Amazon S3 bucket, you must ensure that the bucket is empty or an error will result.
        // If you have a versioned bucket, you must also delete any versioned objects associated with the bucket.
        ListObjectVersionsRequest listObjectVersionsRequest = ListObjectVersionsRequest.builder()
                .bucket(bucketName)
                .build();
        ListObjectVersionsResponse listObjectVersionsResponse = s3Client.listObjectVersions(listObjectVersionsRequest);
        listObjectVersionsResponse.versions()
                .stream()
                .peek(objectVersion -> System.out.println("versionid: " + objectVersion.versionId() +
                        " key: " + objectVersion.key()))
                // create a DeleteObjectRequest
                .map(objectVersion -> DeleteObjectRequest.builder()
                        .bucket(bucketName)
                        .key(objectVersion.key())
                        .versionId(objectVersion.versionId())
                        .build())
                // delete objets one by one
                .forEach(deleteObjectRequest -> s3Client.deleteObject(deleteObjectRequest));

        // once all the objects in a bucket deleted successfully, we can delete empty bucket.
        return deleteEmptyBucket(bucketName);
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
        return false;
    }
 }

The AWS SDK for Java throws a software.amazon.awssdk.services.s3.model.S3Exception when deleting a bucket that contains versioned objects. This exception highlights the requirement to delete all versions within the bucket before deletion can be successful.(alert-error)

Removing S3 Website Bucket Configuration Using S3Client

Amazon S3 can serve as a static website host. While you cannot directly delete a bucket hosting a static website, you can remove its static website configuration.

If the specified bucket doesn't exist, Amazon S3 returns a 404 error code. Upon successfully deleting the bucket's static website configuration, it returns a 200 success code.

 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.*;

 public boolean deleteBucketWebsite(String bucketName) {
    try {
        DeleteBucketWebsiteRequest deleteBucketWebsiteRequest = DeleteBucketWebsiteRequest.builder()
                .bucket(bucketName)
                .build();
        DeleteBucketWebsiteResponse deleteBucketWebsiteResponse = s3Client.deleteBucketWebsite(deleteBucketWebsiteRequest);
        boolean successful = deleteBucketWebsiteResponse.sdkHttpResponse()
                .isSuccessful();
        System.out.println("Is S3 Website Bucket Deleted : " + successful);
        return successful;
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
        return false;
    }
 }

Conclusion

This tutorial has provided a foundational understanding of Amazon S3 bucket operations using the Spring Cloud AWS library. We've explored essential tasks like creating buckets, listing their contents, and effectively deleting them, both with and without versioning enabled.

Additionally, we've delved into the process of removing static website configurations from buckets and have become familiar with common S3Exception scenarios that may arise during development.

Want to know how to efficiently store your files in the cloud? Explore our guide on (getButton) #text=(uploading files to Amazon S3) #icon=(link) #color=(#35a576)

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

How to Create Presigned URLs in Spring Cloud AWS for S3 Storage

2024-09-21T16:58:00.000+05:30

Amazon S3 buckets are initially private, meaning they cannot be accessed publicly. To temporarily grant others access to specific objects, you can create S3 presigned URLs.

These URLs provide limited, time-bound access to the object, allowing you to share the content securely for a specific period

Presigned URLs can be created for both downloads (using the GET HTTP method) and uploads (using the PUT HTTP method).

This allows you to securely share files for downloading or accepting uploads from others within a specified timeframe

In this tutorial, we'll demonstrate how to grant temporary access to S3 bucket objects using Spring Cloud AWS S3 within a Spring Boot application.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Dependencies

Gradle

ext {
    springCloudAwsVersion = '3.0.0'
}

dependencies {
    // spring cloud aws BOM(Bill of materials)
    implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
    // AWS launched a high level file transfer utility, called Transfer Manager and a CRT based S3 client.
    // The starter automatically configures and registers a software.amazon.awssdk.transfer.s3.S3TransferManager bean if the following dependency is added to the project:
    implementation 'software.amazon.awssdk:s3-transfer-manager'
    //Transfer Manager works the best with CRT S3 Client. To auto-configure CRT based S3AsyncClient add following dependency to your project:
    implementation 'software.amazon.awssdk.crt:aws-crt'
}

Maven

<dependencies>
    <dependency>
        <groupId>io.awspring.cloud</groupId>
        <artifactId>spring-cloud-aws-starter-s3</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3-transfer-manager</artifactId>
    </dependency>
	
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws-dependencies</artifactId>
            <version>{project-version}</version>
            <type>pom</type>
            <scope>import</scope>
       </dependency>
    </dependencies>
</dependencyManagement>

Configuration and Credentials

Before diving into bucket operations, let's configure our Spring Boot application to interact with AWS S3. Spring Cloud AWS offers auto-configuration for S3Client, S3TransferManager, and S3Template, making setup a breeze.

Here's what you need to add to your application.properties file:

 # s3 Configuration
 spring.cloud.aws.credentials.access-key=
 spring.cloud.aws.credentials.secret-key=
 # Configures endpoint used by S3Client, I'm woorking in the Asia Pacific (Mumbai) hence, I've configure ap-south1 region and endpoint
 spring.cloud.aws.s3.endpoint=https://s3.ap-south-1.amazonaws.com
 spring.cloud.aws.region.static=ap-south-1

IAM Permissions for Creating Presigned URLs

To follow this tutorial, your IAM user must have the s3:GetObject and s3:PutObject permissions.

You might be wondering why these permissions are necessary if presigned URLs are used to grant access to non-S3 bucket owners. The answer is that presigned URLs can only be generated by someone who already has access to the underlying object.

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DownloadObject",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::*/*"
        },
        {
            "Sid": "UploadObject",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::*/*"
        }
    ]
 }

Creating Presigned URLs for GET Requests with S3Presigner

Let's create our first presigned URL for a GET request using the S3Presigner. Before we begin, ensure you have an active S3 bucket and an object within it.

In this example, I'll use a bucket named "learnjavaskills" with an index.html file. We'll generate a URL to download this file.

 import io.awspring.cloud.s3.S3Exception;
 import software.amazon.awssdk.services.s3.model.GetObjectRequest;
 import software.amazon.awssdk.services.s3.presigner.S3Presigner;
 import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
 import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;

 public URL createPreSignedUrlUsingS3PresignerForGetRequest(String bucketName, String key) {
    try {
        GetObjectRequest objectRequest = GetObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();

        GetObjectPresignRequest objectPresignRequest = GetObjectPresignRequest.builder()
                .signatureDuration(Duration.ofMinutes(10)) // URL will expire after 10 minutes
                .getObjectRequest(objectRequest)
                .build();
        PresignedGetObjectRequest presignedGetObjectRequest = s3Presigner.presignGetObject(objectPresignRequest);

        URL url = presignedGetObjectRequest.url();
        System.out.println("presigned url : " + url.toString());
        return url;
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
        return null;
    }
 }

• S3Presigner : is an interface that allows you to sign an S3 SdkRequest, enabling it to be executed without requiring additional authentication from the caller.

• GetObjectRequest : Retrieves objects from Amazon S3. To use the GET method, you must have READ access to the object. If you grant READ access to the anonymous user, you can retrieve the object without requiring an authorization header.

• GetObjectPresignedRequest : A request to create a presigned URL for a GetObjectRequest, allowing it to be executed at a later time without requiring additional signing or authentication.

Downloading Files Using Presigned URLs with Postman

Once you have the presigned URL, let's use Postman to download the object from the S3 bucket. As you'll see in the response, you can successfully read the index.html file without providing any credentials.

Presigned URL Expiration Responses

The presigned URL we created has a validity of only 10 minutes. If a user attempts to download the file after this interval, Amazon S3 will deny access

Creating Presigned URLs for GET Requests with S3Template

S3Template is a high-level API provided by Spring Cloud AWS that simplifies interactions with Amazon S3. It offers a convenient way to perform common operations like creating, deleting, and managing buckets and objects.

S3Template significantly simplifies the process of creating presigned URLs. By calling the createSignedGetUrl method, you can easily generate a presigned URL for a specified bucket, object key, and duration, allowing you to securely share access to your S3 objects

 import io.awspring.cloud.s3.S3Exception;
 import io.awspring.cloud.s3.S3Template;

 public URL createPreSignedUrlUsingS3TemplateForGetRequest(String bucketName, String key) {
    try {
        return s3Template.createSignedGetURL(bucketName, key, Duration.ofMinutes(10));
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
        return null;
    }
 }

Downloading Files Using Presigned URLs with Postman

I've created a presigned URL for an image. Let's verify its functionality by downloading the image.

Creating Presigned URLs for PUT Requests with S3Presigner

Similarly, we can create a presigned URL for uploading objects to Amazon S3. In this case, we'll need to use a PUT request.

Let's explore how to create a presigned URL for a PUT request using S3Presigner.

To create a presigned URL for uploading a file, we can use the presignPutObject method of the S3Presigner interface.

 import io.awspring.cloud.s3.S3Exception;
 import software.amazon.awssdk.services.s3.model.PutObjectRequest;
 import software.amazon.awssdk.services.s3.presigner.S3Presigner;
 import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
 import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest;

 public URL createPreSignedUrlUsingS3PresignerForPutRequest(String bucketName, String key) {
    try {
        PutObjectRequest putObjectRequest = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(key)
                .build();
        PutObjectPresignRequest putObjectPresignRequest = PutObjectPresignRequest.builder()
                .signatureDuration(Duration.ofMinutes(10)) // URL will expire after 10 minutes
                .putObjectRequest(putObjectRequest)
                .build();

        PresignedPutObjectRequest presignedPutObjectRequest = s3Presigner.presignPutObject(putObjectPresignRequest);
        URL url = presignedPutObjectRequest.url();
        System.out.println("presigned url : " + url.toString());
        return  url;
    } catch (S3Exception s3Exception) {
        s3Exception.printStackTrace();
        return null;
    }
 }

• PutObjectRequest : Uploads a new object to the specified Amazon S3 bucket. It can also optionally upload object metadata and set a canned access control policy for the new object.

• PutObjectPresignRequest : A request to pre-sign a PutObjectRequest so that it can be executed at a later time without requiring additional signing or authentication.

Uploading Files Using Presigned URLs with Postman

Let's verify if the upload presigned URL works by using Postman. Ensure that you set the HTTP method to PUT.

Creating Presigned URLs for PUT Requests with S3Template

Spring Cloud AWS simplifies the process of creating presigned URLs for PUT requests.

By using the createSignedPutUrl method of S3Template, you can easily generate a presigned URL for uploading objects to your S3 bucket, specifying the bucket name, object key, and desired duration of validity.

 import io.awspring.cloud.s3.S3Exception;
 import io.awspring.cloud.s3.S3Template;

 public URL createPreSignedUrlUsingS3TemplateForPutRequest(String bucketName, String key) {
	try {
            return s3Template.createSignedPutURL(bucketName, key, Duration.ofMinutes(10));
	} catch (S3Exception s3Exception) {
		s3Exception.printStackTrace();
            return null;
	}
 }

Conclussion

In this blog tutorial, we've explored the concept of presigned URLs and their significance in securely sharing access to S3 objects. We've learned how to create presigned URLs for both GET (download) and PUT (upload) requests using both S3Presigner and S3Template.

By leveraging presigned URLs, you can grant temporary access to specific S3 objects, enabling you to share files or data securely with external parties without compromising your account's security.

This is particularly useful in scenarios where you need to provide controlled access to sensitive information or collaborate with external teams.

Remember to carefully manage the duration of your presigned URLs to ensure that access is limited to the necessary timeframe.

By following the guidelines and examples provided in this tutorial, you can effectively utilize presigned URLs to enhance the security and flexibility of your S3-based applications.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

How to Use AWS DynamoDB BatchWriteItem and BatchGetItem in Java Spring Boot Applications | Learn Java Skills

2024-09-02T20:26:00.000+05:30

AWS DynamoDB's BatchWriteItem and BatchGetItem operations offer efficient ways to interact with multiple items in a single request, enhancing performance and reducing network overhead compared to individual item operations.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Interacting with DynamoDB Using Java

The AWS SDK for Java provides interface like DynamoDbClient and DynamoDbAsyncClient to interact with the DynamoDB database. These clients offer methods for various DynamoDB operations, including getItem, putItem, batchGetItem, and batchWriteItem

Spring Boot simplifies DynamoDB interaction by auto-configuring the DynamoDbClient based on settings in your application properties file when using the Spring Cloud AWS Starter DynamoDB

Dependencies: Gradle

 // spring cloud aws BOM(Bill of materials)
 implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
 // dynamoDB
 implementation 'io.awspring.cloud:spring-cloud-aws-starter-dynamodb'

Setting Up DynamoDB for Your Application: application.properties

If you're new to Spring Cloud AWS and DynamoDB, I recommend reading our (getButton) #text=(Spring Boot, DynamoDB, and Spring Cloud AWS: A Step-by-Step Tutorial) #icon=(link) #color=(#35a576), which will help you build up your project and get deep into the CRUD (Create, Read, Update, and Delete) operations.

 spring.application.name=spring-cloud-aws-dynamodb

 # DynamoDB connection
 spring.cloud.aws.credentials.access-key=
 spring.cloud.aws.credentials.secret-key=
 # Choose your region according to your table. Because I have a DynamoDB table in the ap-south-1 area, I've set the endpoint and region.
 spring.cloud.aws.region.static=ap-south-1
 spring.cloud.aws.endpoint=https://dynamodb.ap-south-1.amazonaws.com

DynamoDB BatchWriteItem: Efficiently Writing Multiple Items to DynamoDB

The BatchWriteItem approach puts or deletes multiple items into one or more tables. A single call to BatchWriteItem may send up to 16MB of data across the network, which includes up to 25 item put or delete operations.

While individual items can be up to 400 KB once saved, it's crucial to remember that an item's representation may be bigger than 400 KB when submitted in DynamoDB's JSON format for the API request.

BatchWriteItem cannot update items. If you run a BatchWriteItem action on an existing item, the operation will overwrite the item's values, making it look as if it was changed. To update things, AWS recommend using the UpdateItem action.(alert-warning)

Java Program to Perform BatchWriteItems Operations on DynamoDB

This Java program demonstrates how to efficiently insert data into multiple DynamoDB tables using the BatchWriteItem operation. We'll create sample data for Orders and products tables and insert them in a single batch for improved performance.

package in.learnjavaskills.springcloudawsdynamodb.service;

import org.springframework.stereotype.Service;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;

import java.time.LocalDate;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@Service
public class BatchItemExample
{
    private final DynamoDbClient dynamoDbClient;

    public BatchItemExample(DynamoDbClient dynamoDbClient) {
        this.dynamoDbClient = dynamoDbClient;
    }

    public void batchWriteItem() {
        Map<String, AttributeValue> productsMap = new HashMap<>();
        productsMap.put("product_id", AttributeValue.fromS("pd_12345"));
        productsMap.put("product_name", AttributeValue.fromS("Laptop"));
        productsMap.put("description", AttributeValue.fromS("14 inch screen laptop"));
        productsMap.put("price", AttributeValue.fromN("49.87"));
        productsMap.put("stock", AttributeValue.fromN("101"));

        PutRequest productsPutRequest = PutRequest.builder().item(productsMap).build();
        WriteRequest productsWriteRequest = WriteRequest.builder().putRequest(productsPutRequest).build();

        Map<String, AttributeValue> ordersMap = new HashMap<>();
        ordersMap.put("order_id", AttributeValue.fromS("od_12345"));
        ordersMap.put("customer_id", AttributeValue.fromS("c_12345"));
        ordersMap.put("product_id", AttributeValue.fromS("pd_12345"));
        ordersMap.put("order_date", AttributeValue.fromS(LocalDate.now().toString()));

        PutRequest orderPutRequest = PutRequest.builder().item(ordersMap).build();
        WriteRequest orderWriteRequest = WriteRequest.builder().putRequest(orderPutRequest).build();

        Map<String, List<WriteRequest>> writeRequestMap = new HashMap<>();
        writeRequestMap.put("products", Arrays.asList(productsWriteRequest));
        writeRequestMap.put("Orders", Arrays.asList(orderWriteRequest));

        BatchWriteItemRequest batchWriteItemRequest = BatchWriteItemRequest.builder().requestItems(writeRequestMap).build();

        BatchWriteItemResponse batchWriteItemResponse = dynamoDbClient.batchWriteItem(batchWriteItemRequest);

        // Implement a retry mechanism for batchWriteItem operations that encounter errors.
        while (!batchWriteItemResponse.unprocessedItems().isEmpty()) {
            Map<String, List<WriteRequest>> unprocessedItems = batchWriteItemResponse.unprocessedItems();
            batchWriteItemRequest
                    = BatchWriteItemRequest.builder().requestItems(unprocessedItems).build();
            batchWriteItemResponse = dynamoDbClient.batchWriteItem(batchWriteItemRequest);
        }
    }
}

Understanding BatchWriteItemRequest

The batchWriteItem() method in the DynamoDBClient allows you to perform multiple write operations (puts or deletes) across one or more tables in a single request. You provide a BatchWriteItemRequest object to specify the items and actions to be performed.

The requestItems() method in a BatchWriteItemRequest class defines the data to be written to DynamoDB. It takes a map where each key is a DynamoDB table name and the corresponding value is a list of WriteRequest operations (put or delete) to be performed on that table.

Handling Unprocessed Items: Retrying BatchWriteItem Operations

unprocessedItems() is a response parameter returned by the DynamoDB BatchWriteItemRequest operation. It contains a list of items that couldn't be processed due to reasons like exceeding provisioned throughput or internal processing errors.

Essentially, it's a mechanism for handling failed write operations. You can retry these unprocessed items in subsequent BatchWriteItemRequest until all items are successfully written to DynamoDB.

If the requested data exceeds these limits, or if there's insufficient table capacity, or an internal error occurs, the operation might return only a portion of the results. In such cases, the system provides information about the UnprocessedKeys, allowing you to retry the operation to retrieve the remaining data.(alert-passed)

When DynamoDB returns unprocessed items in a batch operation, it's essential to retry these items. However, to avoid overwhelming the system and increasing the likelihood of success, implement an exponential backoff strategy. This means progressively increasing the delay between retries. By doing so, you reduce the chance of encountering throttling issues and improve the overall reliability of your batch operations.(alert-error)

Understanding WriteRequest: Building Blocks for Batch Write Operations

A WriteRequest is a container for specifying either a PutItem or DeleteItem operation within a BatchWriteItem request.

You can only perform one of these operations (Put or Delete) per WriteRequest. If you need to perform both, you'll need separate WriteRequest objects. Each WriteRequest contains either a PutRequest or a DeleteRequest object.

PutRequest: Inserting or Updating Items

If you want to insert or update an item, you use a PutRequest object within your WriteRequest. A PutRequest contains an Item method, which is a map representing the item to be put into the table.

DeleteRequest: Removing Items from DynamoDB

If you want to delete an item, you use a DeleteRequest object within your WriteRequest. A DeleteRequest contains a Key method, which is a map representing the primary key of the item to be deleted.

DynamoDB BatchGetItems: Efficiently Retrieve Multiple Items

You can use the BatchGetItem operation to fetch data from one or more tables. Specify the items you need by their primary keys, and the operation will return their corresponding attributes.

A single BatchGetItem operation can fetch up to 100 items with a maximum data size of 16 MB.

Java Program to Perform BatchGetItem Operations on DynamoDB

This Java program demonstrates how to efficiently fetch data from two DynamoDB tables, Orders and products, using the BatchGetItem operation. By specifying multiple items and their primary keys within a single request, we can optimize data retrieval performance compared to individual GetItem operations.

package in.learnjavaskills.springcloudawsdynamodb.service;

import org.springframework.stereotype.Service;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.*;

import java.time.LocalDate;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@Service
public class BatchItemExample
{
    private final DynamoDbClient dynamoDbClient;

    public BatchItemExample(DynamoDbClient dynamoDbClient) {
        this.dynamoDbClient = dynamoDbClient;
    }
    
    public void batchGetItem() {

        Map<String, AttributeValue> productsKeys = new HashMap<>();
        productsKeys.put("product_id", AttributeValue.fromS("pd_12345"));
        productsKeys.put("product_name", AttributeValue.fromS("Laptop"));

        Map<String, AttributeValue> ordersKeys = new HashMap<>();
        ordersKeys.put("order_id", AttributeValue.fromS("od_12345"));
        ordersKeys.put("customer_id", AttributeValue.fromS("c_12345"));

        Map<String, KeysAndAttributes> requestItems = new HashMap<>();
        KeysAndAttributes productsKeysAndAttributes = KeysAndAttributes.builder().consistentRead(false)
                .keys(productsKeys).build();
        requestItems.put("products", productsKeysAndAttributes);

        KeysAndAttributes ordersKeysAndAttributes  = KeysAndAttributes.builder()
                .keys(ordersKeys)
                .build();
        requestItems.put("Orders", ordersKeysAndAttributes);


        BatchGetItemRequest batchGetItemRequest = BatchGetItemRequest.builder().requestItems(requestItems).build();
        BatchGetItemResponse batchGetItemResponse = dynamoDbClient.batchGetItem(batchGetItemRequest);
        Map<String, List<Map<String, AttributeValue>>> responses = batchGetItemResponse.responses();
        responses.forEach( (table, itemsList) -> {
            System.out.println("table name: " + table);
            itemsList.forEach( attributesAndValues -> {
                attributesAndValues.forEach( (key, value) -> {
                    System.out.println("attributes name : " + key + ", attributes value: " + value.s());
                });
            });
        } );

        // Implement a retry mechanism for batchGetItem operations that encounter errors.
        while(!batchGetItemResponse.unprocessedKeys().isEmpty()) {
            Map<String, KeysAndAttributes> unprocessedKeys = batchGetItemResponse.unprocessedKeys();
            batchGetItemRequest = BatchGetItemRequest.builder().requestItems(unprocessedKeys).build();
            batchGetItemResponse = dynamoDbClient.batchGetItem(batchGetItemRequest);
            responses = batchGetItemResponse.responses();
            responses.forEach( (table, itemsList) -> {
                System.out.println("table name: " + table);
                itemsList.forEach( attributesAndValues -> {
                    attributesAndValues.forEach( (key, value) -> {
                        System.out.println("attributes name : " + key + ", attributes value: " + value.s());
                    });
                });
            } );
        }
    }
}

Understanding BatchGetItemRequest

The batchGetItem() method in the DynamoDBClient allows you to efficiently retrieve multiple items from one or more tables in a single operation. To specify the items you want to fetch, you create a BatchGetItemRequest object.

The requestItems() method within a BatchGetItemRequest specifies the tables and items to retrieve. It takes a map where the keys are table names, and the values are objects containing information about the items to fetch from that table, such as their primary keys and any projection expressions.

Conclusion

By effectively utilizing the BatchWriteItem and BatchGetItem operations, you can significantly enhance the efficiency and performance of your DynamoDB interactions. By understanding the core concepts, including WriteRequest, PutRequest, DeleteRequest, and handling unprocessed items, you're well-equipped to optimize your data manipulation strategies.

To learn more about DynamoDB transactions and how to use TransactWriteItems and TransactGetItems, check out my latest blog post on (getButton) #text=(DynamoDB Transactions with TransactWriteItems and TransactGetItems) #icon=(link) #color=(#35a576)

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

Mastering DynamoDB Data Retrieval: Advanced Scan and Query Operations with Spring Boot and Spring Cloud AWS | Learn Java Skills

2024-05-11T20:24:00.005+05:30

Amazon DynamoDB, a NoSQL database service, offers unparalleled scalability and flexibility for modern applications. But efficiently retrieving data requires understanding its core read operations: queries and scans.

This tutorial dives deep into using Spring Cloud AWS to seamlessly interact with DynamoDB and harness the power of both queries and scans.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Mastering Scans: Unleashing DynamoDB's Power for Comprehensive Data Retrieval

Amazon DynamoDB's Scan operation reads all items from a table or secondary index. By default, a scan operation retrieves all data attributes for each item in a table or index.

You can use the ProjectionExpression parameter to restrict Scan from returning all attributes. Scan always produces a set of results. If no matching items are discovered, the result set is blank.

DynamoDB's Scan operation fetches data in chunks of 1 MB at most. To refine the results and avoid unnecessary data transfer, you can leverage filter expressions that narrow down the items after they're retrieved but before they're delivered.

Scanning DynamoDB with Spring Cloud AWS: Code Example

Ready to unleash the full power of your DynamoDB table? We'll begin by exploring Spring Cloud AWS's Scan operation to extract all its data.

 public PageIterable<Movie> scanMovieTable() {
     PageIterable<Movie> moviePageIterable = dynamoDbTemplate.scanAll(Movie.class);
     return moviePageIterable;
 }

Spring Cloud AWS provides a convenient method called scanAll within its DynamoDbTemplate class. This method takes a single argument - the name of the DynamoDB table you want to scan. In our example, we'll use the Movie class, which represents our data model, to retrieve all items from the specified table.

 @Test
 void scanTable()
 {
     PageIterable<Movie> moviePageIterable = movieService.scanMovieTable();
     if (Objects.nonNull(moviePageIterable))
     {
         SdkIterable<Movie> items = moviePageIterable.items();
         if (Objects.nonNull(items))
             items.forEach(System.out :: println);
     }
 }

Targeting Your Data: Filtering DynamoDB Scans with Expressions

Filter expressions act as a powerful tool to streamline your scans by specifying conditions that retrieved items must meet. This significantly reduces the data volume returned, improving efficiency and focusing on relevant information.

 public PageIterable<Movie> scanMovieTableWithFilterExpression(String movieTitleBeginWith) {

    // Filter expression is nothing but, It's just like a where clause of SQL. Here we are only selecting whose
    // movie_title begin with the given-movie-title(movieTitleBeginWith) like a LIKE operation of SQL.
    Expression expression = Expression.builder()
        .expression("begins_with(movie_title, :movie_title)")
        .putExpressionValue(":movie_title", AttributeValue.fromS(movieTitleBeginWith))
        .build();

    ScanEnhancedRequest scanEnhancedRequest = ScanEnhancedRequest.builder()
        .filterExpression(expression) // filter-expression
        .build();
    PageIterable<Movie> scan = dynamoDbTemplate.scan(scanEnhancedRequest, Movie.class);
    return scan;
 }

DynamoDbTemplate offers a scan method that takes two arguments: a ScanEnhancedRequest object and the class representing your DynamoDB table (in this case, the Movie class).

This approach allows you to specify filtering criteria within the ScanEnhancedRequest for targeted data retrieval.

Expressions Explained: A Powerful Tool for Data Control

Expressions in DynamoDB act as a blueprint for interacting with your data. They can be used for filtering, sorting, and other operations

The expression method of the Expression class allows you to specify a filter expression as a string. This string can contain placeholders for the actual values you want to filter on. You then use the putExpressionValue method to define these placeholder values

Streamlining Scans: Retrieving Only Relevant Data with Projection Expressions

For those familiar with SQL, retrieving only specific columns from a table is common practice.

DynamoDB offers a similar functionality called projection expressions. This allows you to select the exact attributes you need during a scan operation, reducing data transfer and improving efficiency.

 public PageIterable<Movie> scanMovieTableWithProjectionExpression() {

    // ProjectionExpression, If we want to select only selected attributes from the dynamoDB then we can utilize
    // the projection expression, Since we are using the Spring cloud AWS, we can utilize the NestedAttributeName
    // to set the desire attributes name just like below.
    List<NestedAttributeName> nestedAttributeNameList = new ArrayList<>();

    // retrieve characters attribute in a result
    nestedAttributeNameList.add(NestedAttributeName.builder()
         .elements("characters")
         .build());

    // data type of attribute directors is a map, here we are only selecting key = dir2 from the directors attribute.
    nestedAttributeNameList.add(NestedAttributeName.builder()
         .elements("directors", "Dir2")
         .build());

    ScanEnhancedRequest scanEnhancedRequest = ScanEnhancedRequest.builder()
         .addNestedAttributesToProject(nestedAttributeNameList) // projection-expression
         .build();

     PageIterable<Movie> scan = dynamoDbTemplate.scan(scanEnhancedRequest, Movie.class);
     return scan;
}

• NestedAttributeName: In DynamoDB, nested attribute names let you specify how to access data within nested structures. These names are built using a list of strings, where each string represents a level deeper in the nesting. For top-level attributes (those not nested), you can simply create a list with a single string element

In this code example, we're focusing on retrieving just two attributes from the DynamoDB table: characters and directors.

Since 'directors' is stored as a map (key-value pairs), we can be even more specific. We can use the projection expression to target only the value associated with a specific key within the 'directors' map. Here, we're interested in retrieving directors where the key is 'Dir2'.

Leveraging Queries for Efficient Data Retrieval in DynamoDB

DynamoDB's Query operation lets you pinpoint items using their primary key values

You must specify the partition key attribute's name as well as a single value. Query returns all objects with the specified partition key value. You may also specify a sort key attribute and use a comparison operator to filter the search results.

Querying DynamoDB with the Partition Key

 public PageIterable<Movie> findMovieByQuery(Long movieId) {
   Key key = Key.builder().partitionValue(movieId).build();
   QueryConditional queryConditional = QueryConditional.keyEqualTo(key);

   QueryEnhancedRequest queryEnhancedRequest = QueryEnhancedRequest.builder()
     .queryConditional(queryConditional)
     .build();

   PageIterable<Movie> moviePageIterable = dynamoDbTemplate.query(queryEnhancedRequest, Movie.class);
   return  moviePageIterable;
 }

• QueryConditional : Defines the parameters that determine which items are returned when querying DynamoDB tables or indexes with the Query operation. These parameters specify the key conditions and any additional filtering criteria.

• QueryEnhancedRequest : Specifies conditions for finding items in DynamoDB tables or indexes with the Query operation.

 @Test
 public void findMovieByQueryUsingMovieId()
 {
     PageIterable<Movie> movieByQuery = movieService.findMovieByQuery(MOVIE_ID);
     if (Objects.nonNull(movieByQuery))
     {
         SdkIterable<Movie> items = movieByQuery.items();
         if (Objects.nonNull(items))
             items.forEach(System.out::println);
     }
 }

Querying DynamoDB with the Partition Key and Sort key

Just like with the partition key, you can define the sort key for your DynamoDB query by setting it in a Key object and passing that object to the QueryConditional class.

 public PageIterable<Movie> findMovieByQuery(Long movieId, String movieTitle) {
   Key key = Key.builder().partitionValue(movieId).sortValue(movieTitle).build();
   QueryConditional queryConditional = QueryConditional.keyEqualTo(key);

   QueryEnhancedRequest queryEnhancedRequest = QueryEnhancedRequest.builder()
     .queryConditional(queryConditional).build();

   PageIterable<Movie> moviePageIterable = dynamoDbTemplate.query(queryEnhancedRequest, Movie.class);
   return  moviePageIterable;
 }

DynamoDB Query Power: Partition Keys and Filter Expressions

While partition and sort keys are powerful for efficient queries, DynamoDB also allows you to search by other attributes when needed.

Many situations require querying DynamoDB based on attributes beyond the partition and sort keys. These queries leverage filter expressions, similar to how Scan operations utilize them. Filter expressions refine the results of a Query operation after the initial search using keys.

 public PageIterable<Movie> findMovieByQuery(Long movieId, Double rating) {

   Key key = Key.builder().partitionValue(movieId).build();
   QueryConditional queryConditional = QueryConditional.keyEqualTo(key);

   Expression expression = Expression.builder()
     .expression("rating >= :rating_value")
     .putExpressionValue(":rating_value", AttributeValue.fromN(rating.toString()))
     .build();

   QueryEnhancedRequest queryEnhancedRequest = QueryEnhancedRequest.builder()
     .queryConditional(queryConditional) // query-condition with partition-key
     .filterExpression(expression) // filter-expression
     .build();

   PageIterable<Movie> moviePageIterable = dynamoDbTemplate.query(queryEnhancedRequest, Movie.class);
   return moviePageIterable;
 }

Streamlining Results: Optimizing Data Retrieval with Projection Expressions

For situations where you only need a subset of data, DynamoDB offers projection expressions. These allow you to fetch only the relevant attributes from the table, reducing data transfer and improving performance.

In the example below, we'll demonstrate how projection expressions work. We're only interested in retrieving two specific attributes: movie_id and movie_title.

Similarly, in the following example, I demonstrate how we may use both filter expression and projection expressions.

 public PageIterable<Movie> findMovieTitle(String genreContain, Long movieId) {

   // it's mandatory to use the key with the filter expression
   Key key = Key.builder().partitionValue(movieId).build();
   QueryConditional queryConditional = QueryConditional.keyEqualTo(key);

   // filter expression, finding only those movie which genre's contain genreContain
   Expression expression = Expression.builder()
     .expression("contains(genre, :genre_contain_value)")
     .putExpressionValue(":genre_contain_value", AttributeValue.fromS(genreContain))
     .build();

   // Selecting only movie_id and movie title attributes only.
   List<NestedAttributeName> nestedAttributeNameList = new ArrayList<>();

   nestedAttributeNameList.add(NestedAttributeName.builder()
     .elements("movie_id")
     .build());

   nestedAttributeNameList.add(NestedAttributeName.builder()
     .elements("movie_title")
     .build());

   QueryEnhancedRequest queryEnhancedRequest = QueryEnhancedRequest.builder()
     .queryConditional(queryConditional)
     .filterExpression(expression) // filter-expression
     .addNestedAttributesToProject(nestedAttributeNameList) // projection-expression
     .build();

   PageIterable<Movie> moviePageIterable = dynamoDbTemplate.query(queryEnhancedRequest, Movie.class);
   return moviePageIterable;
 }

Conclusion

Spring Cloud AWS provides a powerful and convenient way to integrate DynamoDB with your Spring Boot applications. By leveraging the Query and Scan operations, you can efficiently retrieve data based on specific criteria or scan the entire table as needed.

We've explored using partition and sort keys for targeted queries, filter expressions for narrowing down results based on additional attributes, and projection expressions for optimizing data retrieval.

With this knowledge, you can unlock the full potential of DynamoDB within your Spring Boot projects, ensuring efficient data access and streamlined application performance.

Optimize your DynamoDB interactions with BatchWriteItem and BatchGetItem, See how: (getButton) #text=(How to Use AWS DynamoDB BatchWriteItem and BatchGetItem in Java Spring Boot Applications) #icon=(link) #color=(#35a576)

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

Keep learning and keep growing.

Spring Boot, DynamoDB, and Spring Cloud AWS: A Step-by-Step Tutorial | Learn Java Skills

2024-04-30T14:03:00.007+05:30

Building high-performance and scalable applications often requires a powerful database solution.

If you're working with a Java Spring project and considering Amazon's robust NoSQL service, DynamoDB, this article is for you! We'll guide you through a seamless integration process using the Spring Cloud AWS.

By following this tutorial, you'll gain the knowledge and tools to leverage DynamoDB's flexibility and speed within your Spring application, taking your project's data storage capabilities to the next level.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

~~toc~~

Dive into DynamoDB: A Scalable NoSQL Powerhouse for Modern Applications

DynamoDB is a NoSQL, key-value database service offered by Amazon Web Services (AWS). Unlike relational databases that rely on predefined schemas, DynamoDB offers a flexible data model that scales seamlessly to accommodate massive datasets and surging traffic

This makes it ideal for applications with unpredictable workloads or those requiring ultra-low latency for data access.

Imagine building a mobile app with millions of users – DynamoDB can effortlessly handle the surge in data traffic, ensuring a smooth and responsive experience for everyone.

Creating a DynamoDB Table with the AWS CLI

Before you can interact with DynamoDB from your Spring Cloud AWS application, you'll need to have a table already set up in your DynamoDB database.

Make sure you have the AWS CLI installed and configured with your AWS access key ID and secret access key. This allows the CLI to interact with your AWS resources, including DynamoDB. You can find instructions for configuration in the AWS documentation

 aws dynamodb create-table --table-name movie \
    --attribute-definitions AttributeName=movie_id,AttributeType=N AttributeName=movie_title,AttributeType=S \
    --key-schema AttributeName=movie_id,KeyType=HASH AttributeName=movie_title,KeyType=RANGE \
    --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

Once you executed the above script, you should see your table details in JSON format in the response, as seen below.

{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "movie_id",
                "AttributeType": "N"
            },
            {
                "AttributeName": "movie_title",
                "AttributeType": "S"
            }
        ],
        "TableName": "movie",
        "KeySchema": [
            {
                "AttributeName": "movie_id",
                "KeyType": "HASH"
            },
            {
                "AttributeName": "movie_title",
                "KeyType": "RANGE"
            }
        ],
        "TableStatus": "CREATING",
        "CreationDateTime": "2024-04-26T10:33:15.733000+00:00",
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 1,
            "WriteCapacityUnits": 1
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:us-east-1:581907371749:table/movie",
        "TableId": "ef4ae555-0d45-4ba5-a846-d78c6098d765",
        "DeletionProtectionEnabled": false
    }
}

Maximum number of tables per account and region.
Any AWS account has an initial quota of 2,500 tables per AWS Region.

If you require more than 2,500 tables for a single account, please contact the Amazon Web Services (AWS) support team to discuss an increase to a maximum of 10,000 tables. For more than 10,000 tables, the recommended best practice is to create numerous accounts, each capable of serving up to 10,000 tables.(alert-passed)

Adding the Spring Cloud AWS DynamoDB dependency

Spring Cloud AWS provides a starter dependency to simplify integrating with DynamoDB.

 ext {
    springCloudAwsVersion = '3.0.0'
 }

 dependencies {
    // spring cloud aws BOM(Bill of materials)
    implementation platform("io.awspring.cloud:spring-cloud-aws-dependencies:${springCloudAwsVersion}")
    // dynamoDB
    implementation 'io.awspring.cloud:spring-cloud-aws-starter-dynamodb'
 }

Configuring DynamoDB with Spring Boot and Spring Cloud AWS (application.properties)

This simple configuration guide explains how to integrate Amazon DynamoDB into your Spring Boot application with Spring Cloud AWS. Details on credentials, region, and connection properties will be included in the application.properties file.

 # DynamoDB connection
 spring.cloud.aws.credentials.access-key=AWS_ACCESS_KEY
 spring.cloud.aws.credentials.secret-key=AWS_SECRET_KEY
 # Choose your region according to your table. Because I have a DynamoDB table in the US-East-1 area, I've set the endpoint and region.
 spring.cloud.aws.region.static=us-east-1 
 spring.cloud.aws.endpoint=https://dynamodb.us-east-1.amazonaws.com

Defining DynamoDB Documents (Model/Entity)

This guide dives into crafting well-structured documents (data models) for your 'Movies' table in DynamoDB.

It clarifies the concept of DynamoDB documents (JSON data structures) and provides best practices for designing them to efficiently store and retrieve movie data within your application.

package in.learnjavaskills.springcloudawsdynamodb.document;

import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbAttribute;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbBean;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbPartitionKey;
import software.amazon.awssdk.enhanced.dynamodb.mapper.annotations.DynamoDbSortKey;

import java.util.List;
import java.util.Map;
import java.util.Set;

@DynamoDbBean
public class Movie
{
    private Long movieId;
    private String movieTitle;
    private Map<String, List<String>> directors;
    private Set<String> characters;
    private Genres genre;
    private Boolean baseOnBook;
    private Double rating;

    public Movie()
    {
    }

    public Movie(Long movieId, String movieTitle, Map<String, List<String>> directors, Set<String> characters, Genres genre, Boolean baseOnBook, Double rating)
    {
        this.movieId = movieId;
        this.movieTitle = movieTitle;
        this.directors = directors;
        this.characters = characters;
        this.genre = genre;
        this.baseOnBook = baseOnBook;
        this.rating = rating;
    }

    @DynamoDbPartitionKey
    @DynamoDbAttribute(value = "movie_id")
    public Long getMovieId()
    {
        return movieId;
    }

    public void setMovieId(Long movieId)
    {
        this.movieId = movieId;
    }

    @DynamoDbSortKey
    @DynamoDbAttribute(value = "movie_title")
    public String getMovieTitle()
    {
        return movieTitle;
    }

    public void setMovieTitle(String movieTitle)
    {
        this.movieTitle = movieTitle;
    }

    @DynamoDbAttribute(value = "directors")
    public Map<String, List<String>> getDirectors()
    {
        return directors;
    }

    public void setDirectors(Map<String, List<String>> directors)
    {
        this.directors = directors;
    }

    public Set<String> getCharacters()
    {
        return characters;
    }

    public void setCharacters(Set<String> characters)
    {
        this.characters = characters;
    }

    public Genres getGenre()
    {
        return genre;
    }

    public void setGenre(Genres genre)
    {
        this.genre = genre;
    }

    public Boolean getBaseOnBook()
    {
        return baseOnBook;
    }

    public void setBaseOnBook(Boolean baseOnBook)
    {
        this.baseOnBook = baseOnBook;
    }

    public Double getRating()
    {
        return rating;
    }

    public void setRating(Double rating)
    {
        this.rating = rating;
    }

    @Override
    public String toString()
    {
        return "Movie{" + "movieId=" + movieId + ", movieTitle='" + movieTitle + '\'' + ", directors=" + directors + ", characters=" + characters + ", genre=" + genre + ", baseOnBook=" + baseOnBook + ", rating=" + rating + '}';
    }
}

Introducing `@DynamoDbBean`: Mapping Your Model to DynamoDB Documents

The @DynamoDbBean annotation is a cornerstone of Spring Cloud AWS for interacting with Amazon DynamoDB.

It acts as a bridge between your application's model classes and DynamoDB documents, allowing you to seamlessly map your data structures to the NoSQL database format.

Key Attributes: @DynamoDbPartitionKey and @DynamoDbSortKey

Within a DynamoDB table, data is organized using primary keys. Spring Cloud AWS provides annotations to define these keys within your model classes:

• @DynamoDbPartitionKey: This annotation marks a property in your model class as the partition key. The partition key is the primary way DynamoDB partitions your data and efficiently retrieves items. It's crucial to choose an attribute with high cardinality (frequent unique values) for optimal performance.

Partition key length
The minimum length of a partition key value is 1 byte. The maximum length is 2048 bytes.(alert-success)

• @DynamoDbSortKey: This annotation identifies a property as the sort key. The sort key further refines how DynamoDB retrieves items within a partition. It's often used for efficient range queries on frequently accessed data sets within a partition.

Sort key length
The minimum length of a sort key value is 1 byte. The maximum length is 1024 bytes.(alert-success)

Mapping Model Properties: @DynamoDbAttribute

The @DynamoDbAttribute annotation plays a vital role in Spring Cloud AWS for DynamoDB. It signifies that a property in your model class corresponds to an attribute within a DynamoDB document. Here's a breakdown of its functionality:

• Marks Properties: It explicitly tells Spring Cloud AWS to map the annotated property to a DynamoDB attribute.

• Optional (Sometimes): By default, if the property name in your model class matches the DynamoDB attribute name, this annotation is not strictly required. As evident in the movie mode class, I have purposefully chosen not to use this annotation for a few of the properties. Spring Cloud AWS will match the DynamoDB attribute with the movie class property name.

• Custom Attribute Names: If the property name differs from the desired DynamoDB attribute name, you can specify the attribute name using the value parameter within the annotation.

Attribute names
In general, an attribute name must be at least one character long, but no greater than 64 KB long.(alert-success)

Performing CRUD Operations on DynamoDB with Spring Cloud AWS

Let's explore at the Create, Read, Update, and Delete (CRUD) operations on the DynamoDB table we created earlier, i.e., the movie, using Spring Cloud AWS.

The DynamoDBTemplate: Your Interface for CRUD Operations

The starter automatically configures and registers a DynamoDbOperations bean, which provides higher-level abstractions for working with DynamoDB. DynamoDbTemplate, a default DynamoDbOperations implementation, is built on top of DynamoDbEnhancedClient and uses annotations provided by AWS.

DynamoDbTemplate allow you to perform CRUD operations (Create, Read, Update, Delete) on your DynamoDB tables in a more convenient and Spring-friendly manner and It may be found in the io.awspring.cloud.dynamodb.DynamoDbTemplate package.

Persisting Movie Data: Saving Movies with DynamoDBTemplate

This section explores how to leverage the DynamoDbTemplate class to seamlessly save movie data within your DynamoDB 'Movies' table. Notably, the save method within DynamoDbTemplate serves as the primary mechanism for persisting your movie information as documents in the DynamoDb database.

 public Long addMovie(Movie movie) {
    if (Objects.isNull(movie))
        throw new DynamoDbDataException("Movie must be non null");

    Movie savedMovie = dynamoDbTemplate.save(movie);
    if (Objects.nonNull(savedMovie) && Objects.nonNull(savedMovie.getMovieId()))
        return savedMovie.getMovieId();
    throw new DynamoDbDataException("Unsaved movie, Please try again.");
 }

 public static final String HARRY_POTTER_AND_THE_DEATHLY_HALLOWS_PART_1 = "Harry Potter and the Deathly Hallows: Part 1";
 public static final long MOVIE_ID = 101L;
 @Autowired private MovieService movieService;

 @Test
 void saveMovie() {
    Map<String, List<String>> directors = new HashMap<>();
    List<String> directorsName = new ArrayList<>();
    directorsName.add("David Yates");
    directors.put("Dir1", directorsName);

    Set<String> characters = new HashSet<>();
    characters.add("Harry Potter");
    characters.add("Hermione Granger");
    characters.add("Ron Weasley");
    characters.add("Draco Malfoy");

    Movie movie = new Movie(MOVIE_ID, HARRY_POTTER_AND_THE_DEATHLY_HALLOWS_PART_1,
		directors, characters, Genres.DRAMA, true, 3.9);

    Long movieId = movieService.addMovie(movie);
    Assertions.assertThat(movieId).isEqualTo(movie.getMovieId());
 }

DynamoDB's maximum item size is 400 KB, which includes both attribute name binary length (UTF-8 length) and attribute value lengths (again, binary length). The attribute name counts toward the size limit.
For example, take an item with two attributes: "movie-name" with the value "Harry Potter and the Philosopher's Stone" and "release-date" with the value "2001". The item's entire size is 67 bytes. (alert-passed)

Retrieving Movie Information: Reading Data with DynamoDBTemplate

This section explores how to efficiently retrieve movie data from your DynamoDB 'Movies' table using the DynamoDbTemplate class.

A core method for retrieving data is the load method. It takes two crucial parameters:

• Class: This specifies the class that represents your movie data model (the class annotated with @DynamoDbBean)

• Key: This parameter is fundamental for pinpointing the specific movie entry you want to fetch. The key should correspond to the primary key defined in your model class using @DynamoDbPartitionKey and optionally @DynamoDbSortKey annotations. By providing the appropriate key values, load efficiently retrieves the corresponding movie document from DynamoDB.

 public Optional<Movie> findMovie(Long movieId, String movieTitle) {
    Key key = Key.builder()
    	.partitionValue(movieId)
        .sortValue(movieTitle)
        .build();
    Movie movie = dynamoDbTemplate.load(key, Movie.class);
    return Optional.ofNullable(movie);
 }

 @Test
 public void findMovie() {
    Optional<Movie> movie = movieService.findMovie(MOVIE_ID, HARRY_POTTER_AND_THE_DEATHLY_HALLOWS_PART_1);
    Assertions.assertThat(movie).isPresent();
 }

Removing Movies: Deleting Items with DynamoDBTemplate

This section tackles deleting movie entries from your DynamoDB 'Movies' table using the DynamoDbTemplate class.

Just like the load() method, the delete() method also takes two objects as input, i.e., the key and the instance of the model class.

 void deleteMovie(Long movieId, String movieTitle) {
    Key key = Key.builder()
    	.partitionValue(movieId)
        .sortValue(movieTitle)
        .build();
    dynamoDbTemplate.delete(key, Movie.class);
 }

 @Test
 public void deleteMovie() {
    movieService.deleteMovie(MOVIE_ID, HARRY_POTTER_AND_THE_DEATHLY_HALLOWS_PART_1);
 }

You may alternatively simply delete entries from the DynamoDB table by calling the overloaded delete method and accepting the data model, as seen in the example below.

 public void deleteMovie(Movie movie) {
     dynamoDbTemplate.delete(movie);
 }

Modifying Movies: Updating Data with DynamoDBTemplate

This section delves into updating existing movie entries within your DynamoDB 'Movies' table using the DynamoDbTemplate class. We'll explore how to leverage the update method of DynamoDbTemplate to make targeted changes to your movie data, ensuring your information remains accurate and up-to-date.

The update method is the primary mechanism for updating data in DynamoDB. It takes a single argument:

• Your Model Class Instance: You pass an instance of your movie data model class (annotated with @DynamoDbBean). The update method recognizes the changes made to the object's properties and translates them into corresponding updates within the DynamoDB document associated with the movie's primary key. This approach simplifies the update process, as you only need to modify the relevant fields in your model object.

 public Long updateMovie(Movie movie) { 
    Movie updatedMovie = dynamoDbTemplate.update(movie);
    if (Objects.nonNull(updatedMovie))
      return updatedMovie.getMovieId();
    return 0L;
 }

 @Test
 public void updateMovie() {
    Map<String, List<String>> directors = new HashMap<>();
    List<String> directorsName = new ArrayList<>();
    directorsName.add("David Yates");
    directors.put("Dir1", directorsName);

    Set<String> characters = new HashSet<>();
    characters.add("Harry Potter");
    characters.add("Hermione Granger");
    characters.add("Ron Weasley");
    characters.add("Draco Malfoy");

    Movie movie = new Movie(MOVIE_ID, HARRY_POTTER_AND_THE_DEATHLY_HALLOWS_PART_1,
        directors, characters, Genres.DRAMA, true, 4.7);

    Long movieId = movieService.updateMovie(movie);
    Assertions.assertThat(movieId).isEqualTo(MOVIE_ID);
 }

Conclusion

This article has explored performing CRUD operations (Create, Read, Update, Delete) on your DynamoDB 'Movies' table using Spring Cloud AWS and the DynamoDbTemplate class. We've equipped you with the tools to effectively manage your movie data within the DnamoDb database.

The complete code for this project is available on my GitHub repository (getButton) #text=(GitHub) #icon=(share) #color=(#000000)

For a deep dive into scan and query operations, this blog is a great resource: (getButton) #text=(Mastering DynamoDB Data Retrieval: Advanced Scan and Query Operations) #icon=(link) #color=(#35a576)

Keep learning and keep growing.

How to create serverless function with spring cloud function, deploy into AWS Lambda and expose rest endpoint using AWS API Gateway | Learn Java Skills

2024-03-03T15:16:00.000+05:30

Now a days, serverless functions are gaining much more popularity because developers don't want to spend valuable time configuring the server themselves or maintaining the operating system patch update manually.

There are three main advantages to using the serverless function (Spring Cloud Function).

No efforts were required for us to setup servers, like software installation, before deploying them into production.

Maintenance of the server, like software patch updates, is eliminated.

Serverless functions are auto-scalable by nature if you deploy them in the cloud, such as AWS, GCP, or Azure.

In this tutorial, you can expect to learn how to build your first serverless function using the Spring cloud function and deploy it into the cloud.

There are many cloud providers that offer serverless function deployment, such as Google (GCP), Azure, or Amazon AWS. In this tutorial, you can expect the end-to-end development of the spring cloud function and deployment into the AWS lambda.

We are not only deploying our cloud function into AWS, but we will also make sure that it is accessible via the Rest endpoint by configuring the AWS API Gateway with our AWS lambda.

~~toc~~

This tutorial is divided into four sections, as follows:

First, we will create a Spring cloud function and run it locally.

Second, in Spring Boot version 3, we are going to build a Spring Cloud function that should be compatible with AWS lambda function deployment.

We will learn how to trigger the AWS lambda function with the AWS API Gateway endpoint.

Fourth, we'll look at how we can configure this spring cloud function in Spring Boot version 2, as well as what adjustments we need to make to the handler setting in the AWS lambda function when compared to Spring Boot version 3.

Spring Cloud Function in local

In this section we will learn how to develope a spring cloud function and run function locally.

Maven Dependency

Thre are two main dependencies need to be added and they are as followed

spring-cloud-starter-function-web
spring-cloud-dependencies

  <?xml version="1.0" encoding="UTF-8"?>
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <parent>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-parent</artifactId>
      <version>3.1.3</version>
      <relativePath/> <!-- lookup parent from repository -->
   </parent>
   <groupId>in.learnjavaskills</groupId>
   <artifactId>spring-cloud-function-local</artifactId>
   <version>0.0.1-SNAPSHOT</version>
   <name>spring-cloud-function-local</name>
   <description>Demo project for Spring Boot</description>
   <properties>
      <java.version>17</java.version>
      <spring-cloud.version>2022.0.4</spring-cloud.version>
   </properties>
   <dependencies>
      <!-- Spring Cloud Function  -->
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-starter-function-web</artifactId>
      </dependency>
      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-test</artifactId>
         <scope>test</scope>
      </dependency>
   </dependencies>
   <dependencyManagement>
      <dependencies>
         <!-- spring cloud dependencies -->
         <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud.version}</version>
            <type>pom</type>
            <scope>import</scope>
         </dependency>
      </dependencies>
   </dependencyManagement>

   <build>
      <plugins>
         <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
         </plugin>
      </plugins>
   </build>

 </project>

The Spring cloud function along with the functional interface introduction.

Only one rest endpoint can be expose by using the Spring cloud function. You can configure many functional interfaces with the bean in your application, but only one should be accessible to the end user at a time.

The Spring Cloud function exposes the rest endpoint using three frequently utilized functional interfaces. This functional interface, which was introduced in Java 8 as part of the java.util.function package, is as follows:

Function Interface

The Function interface has two arguments that we must declare when utilizing it: T and R. The T parameter indicates the function's request type, while the R parameter determines the function's return type.

 @FunctionalInterface
 public interface Function<T, R> {
 }

The abstract method in the Function interface that we must override when using the Function interface is the apply() method.

The apply() method expects to return the result with the same return data type that we defined when we declared the generic type in the function interface. It accepts one argument of the data type that we specified when we created the function interface.

R apply(T t);~~(code-box)~~

Consumer Interface

The consumer interface, as the name suggests, is used to consume data and does not return any data. The only generic available in the user interface is T.

 @FunctionalInterface
 public interface Consumer<T> {
 }

The consumer interface has one abstract method, accept(). The accept method takes one parameter of type specified as the consumer interface's generic type.

void accept(T t);~~(code-box)~~

Supplier Interface

When we wish to get some data without providing any arguments as input, we utilize the supplier interface. When using the supplier interface, we only need to decide on one generic type, T, which specifies the function's return type.

 @FunctionalInterface
 public interface Supplier<T> {
 }

The Supplier interface has one abstruct function, get(). The get() method must return data of the same generic T type that has been defined in the Suppier interface's generic type.

T get();~~(code-box)~~

Developing a Rest API using the Spring cloud function

I'm developing an API that will be responsible for counting the amount of words in a given String. I've created one java record (Response) to transmit the API response. First, let's make a Response record.

If you are using Java 8 or Java 11, you may substitute POJO for the java record.

 package in.learnjavaskills.springcloudfunctionlocal;

 public record Response(int status, String message, Long wordCount)
 {}

Create a @Bean to expose the Rest Endpoint.

After we've finished developing the response record file, we can start working on a bean to expose the rest enpoint.

 package in.learnjavaskills.springcloudfunctionlocal;

 import org.springframework.boot.SpringApplication;
 import org.springframework.boot.autoconfigure.SpringBootApplication;
 import org.springframework.context.annotation.Bean;

 import java.util.Objects;
 import java.util.function.Function;

 @SpringBootApplication
 public class SpringCloudFunctionLocalApplication
 {

   public static void main(String[] args)
   {
     SpringApplication.run(SpringCloudFunctionLocalApplication.class, args);
   }

   @Bean(name = "find-word-count")
   public Function<String, Response> findWordCount()
   {
     return request ->
     {
       if (Objects.nonNull(request) && Objects.nonNull(request) && !request.isEmpty())
       {
         if (request.contains(" "))
         {
           long wordCount = request.split(" ").length;
           return new Response(200, "success", wordCount);
         }
       }
       return new Response(409, "fail", 0L);
     };
   }
 }

Testing rest enpoind

Let's put it to the test now. Use curl to execute the above enpoint.

curl -H "Content-Type: application/json" localhost:8080/find-word-count -d "Learn Java Skills"~~(code-box)~~

The output is shown below.

{"status":200,"message":"success","wordCount":3} ~~(code-box)~~

How do you write a Spring Cloud function in Spring Boot 3 and publish it to AWS Lambda?

Till, Now we are able to develop the spring cloud function, which we can also run on our local machine. But this function is not yet ready to deploy into AWS because we do need to update the dependencies and code to make it compatible with the AWS lambda function.

Let's develop a serverless function with the Spring cloud function and deploy it to AWS Lambda. This time, we'll write a method that will reverse the given string.

AWS Lambda Dependencies

The libraries listed below are necessary in our project to develop the serverless function (Spring Cloud Function) and deploy it to AWS Lambda.

spring-cloud-function-adapter-aws
aws-lambda-java-events
aws-lambda-java-core
spring-cloud-dependencies

pom.xml

 <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.1.3</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>
  
  <groupId>in.learnjavaskills</groupId>
  <artifactId>spring-cloud-function-aws-lambda</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <name>spring-cloud-function-aws-lambda</name>
  <description>Demo project for Spring Boot</description>
  
  <properties>
    <java.version>17</java.version>
    <spring-cloud.version>2022.0.4</spring-cloud.version>
    <aws-lambda-events.version>3.9.0</aws-lambda-events.version>
    <aws-lambda-java-core-version>1.1.0</aws-lambda-java-core-version>
  </properties>
  
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-function-adapter-aws</artifactId>
    </dependency>

    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-lambda-java-events</artifactId>
      <version>${aws-lambda-events.version}</version>
    </dependency>

    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-lambda-java-core</artifactId>
      <version>${aws-lambda-java-core-version}</version>
      <scope>provided</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-dependencies</artifactId>
        <version>${spring-cloud.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Thin layout plugin

Before we can deploy our spring cloud function into the AWS lambda function, we must first build a thin jar of our project.

Why do we require a thin jar?

Spring Boot applications come with lots of libraries, and AWS Lambda provides a layer where we can upload our common library, and later, when establishing an AWS function, we can add those layers to our function.

We only provide our source code by producing a thin jar. We will additionally generate a bundled jar containing all of the libraries required to run our AWS Lambda layer.

Add the spring-boot-thin-layout and maven-shade-plugin plugins to the build tag to generate the thin and bundle jars.

 <build>
    <plugins>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-deploy-plugin</artifactId>
        <configuration>
          <skip>true</skip>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <dependencies>
          <!-- building thin jar for deploying into the AWS lambda, because dependencies
            are required at the runtime in the AWS lambda -->
          <dependency>
            <groupId>org.springframework.boot.experimental</groupId>
            <artifactId>spring-boot-thin-layout</artifactId>
            <version>${wrapper.version}</version>
          </dependency>
        </dependencies>
        <configuration>
          <excludes>
            <exclude>
              <groupId>org.projectlombok</groupId>
              <artifactId>lombok</artifactId>
            </exclude>
          </excludes>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <configuration>
          <createDependencyReducedPom>false</createDependencyReducedPom>
          <shadedArtifactAttached>true</shadedArtifactAttached>
          <shadedClassifierName>aws</shadedClassifierName>
        </configuration>
      </plugin>

    </plugins>

  </build>

Write a Spring Cloud function to reverse a given string.

We'll start by writing a function and annotating it with the @Bean annotation. As a result, it will be triggered when we run the AWS lambda code.

 package in.learnjavaskills.springcloudfunctionawslambda;

 import in.learnjavaskills.springcloudfunctionawslambda.dto.GatewayRequest;
 import in.learnjavaskills.springcloudfunctionawslambda.dto.GatewayResponse;
 import org.springframework.boot.SpringApplication;
 import org.springframework.boot.autoconfigure.SpringBootApplication;
 import org.springframework.context.annotation.Bean;

 import java.util.Objects;
 import java.util.function.Function;

 @SpringBootApplication
 public class SpringCloudFunctionAwsLambdaApplication 
 {
   public static void main(String[] args) 
   {
     SpringApplication.run(SpringCloudFunctionAwsLambdaApplication.class, args);
   }

   @Bean
   public Function<GatewayRequest, GatewayResponse> reverseTextFunction()
   {
     return gatewayRequest -> {
       if (Objects.nonNull(gatewayRequest) && Objects.nonNull(gatewayRequest.getText()))
       {
         String reverseText = new StringBuffer(gatewayRequest.getText()).reverse().toString();
         return new GatewayResponse(200, reverseText, "Success");
       }
       return new GatewayResponse(409, null, "fail");
     };
   }
 }

As you can see from the code above, I used GatewayRequest and GatewayResponse to accept the user request and return the result. In our code, let us create both POJOs and record.

 public class GatewayRequest
 {
   private String text;

   public String getText()
   {
     return text;
   }
   // Getter and Setter, toString, equal and hashcode methods
 }

 package in.learnjavaskills.springcloudfunctionawslambda.dto;

 public record GatewayResponse(int status, String reverseText, String message)
 {}

For the GatewayResponse, I have use a record. You can utilize Java Pojo in your Spring Cloud function if you are running Java 8 or Java 11.

Create packages for AWS Lambda deployment.

Let's create a package by running the following command:

mvn clean install ~~(code-box)~~

After successfully building the package, you should see the following jar in the targer folder:

We are only interested in spring-cloud-function-aws-lambda-0.0.1-SNAPSHOT.jar and spring-cloud-function-aws-lambda-0.0.1-SNAPSHOT-aws.jar for the time being. Which will be used in the AWS Lambda function and AWS Lambda Layer.

Set up the AWS Lambda Function with the Spring Cloud Function.

You must have an active AWS account before attempting this activity. If you don't already have an AWS account, you may sign up for a 12-month free tier account from here.

Log into your AWS Lambda account, select the nearest region, and then search for AWS Lambda in the search field, as seen in the image below.

I'm working in the Mumbai region for this activity.

After successfully completing the lambda search, you will be directed to the AWS lambda function's home page, as seen below.

To create a new lambda function, click the create a function button.

You'll now be asked to choose a function creation option. We must select the author from scratch option from the screen given below because we will be deploying our own Spring Cloud service.

Let's fill out the Basic Information box with the lambda function name and runtime version.

I named it spring-cloud-function-with-spring-boot-version-3 because I used Spring Boot version 3 instead of Spring Boot version 2 since Spring Boot version 2 has various setups, which I'll illustrate later.

Because I use open JDK 17 in order to build the Spring cloud function on my machine, the runtime version is Java 17. You can change the Java version to match your JDK version.

Once you've completed all of the steps, you may use the create function button to create your first AWS lambda function.

When you use the create function button, your first AWS lambda has been successfully created. You can confirm this by looking at the screen below.

Are all of our AWS Lambda configurations complete?

Surprisingly, we have yet to deploy our spring cloud function code into the AWS Lambda function.

Remember that we made two jars. Spring-cloud-function-aws-lambda-0.0.1-SNAPSHOT.jar was one, and spring-cloud-function-aws-lambda-0.0.1-SNAPSHOT-aws.jar was the other.

It's time to deploy one of those jars into the lambda function. The spring-cloud-function-aws-lambda-0.0.1-SNAPSHOT.jar package jar is our thin jar, which we are now putting into our AWS Lambda function.

Scroll down to the code portion of the AWS Lambda function, where you will find the upload from button, as seen in the image below. To deploy spring-cloud-function-aws-lambda-0.0.1-SNAPSHOT.jar, click the upload from button.

In the aws lambda function, you have two options for uploading your artifact or jar. You can either zip your artifact or jar before uploading it to the lambda function, or you can immediately upload your jar.

Second, you can upload your artifact or jar into the AWS S3 bucket and then selecting it into the AWS lambda function to upload it.

In our example, we are uploading our jar directly into the AWS Lambda function by using the.zip or.jar option.

An upload box will appear, allowing you to select your jar, which will be deployed into the AWS Lambda function. Click the upload button, navigate to the jar in your file explorer, and then click the save button.

Congratulation, Your thin jar was successfully uploaded to the AWS lambda function. You may validate this by looking at the screen below on your AWS Lambda Function page:

Adding a layer to the AWS layer or installing a dependent Spring cloud function library

Is our Aws Lambda function suitable for testing?

Unfortunately, it is not. We still need to deploy our spring cloud function library into the AWS layer and connect it to our AWS lambda function. Till now we have only deployed our code into the AWS lambda function.

Let's make a layer and insert our jar into it.

Hit the three horizontal lines in the top left sidebar to expand the side bar menu, and then select the layer as shown in the screenshot below.

You'll now be taken to the layer's main screen. To begin generating a layer, click the create layer button, as seen below.

To upload the spring-cloud-function-aws-lambda-0.0.1-SNAPSHOT-aws.jar into the layer, zip the artifact into the java/lib folder.

Complete the layer configuration form. You can enter any name in the name text box, however for this example, I'm using spring-cloud-function-library.

Even if it is optional, don't forget to fill out the description. It will assist you in determining which libraries are included in this layer.

Upload your zip file, choose a compatible runtime (for us, Java 17), and click the Create button.

The AWS Lambda function allows us to upload a jar with a maximum size of 50 MB, and if the jar size is larger than or equivalent to 10 MB, we should use AWS S3.(alert-success)

Hurray! We have finished developing a layer in AWS lambda. Look at the screen below to confirm your layer creation.

Attached layer into the AWS Lambda Function

It's finally time to connect this layer to the AWS lambda function.

Open your lambda function, then scroll down until you find the layers section. To add a layer to this AWS lambda function, locate the add layer button.

The Add Layer dialog box will display. Select the Custom Layer radio button under the Choose Layer section, and then select the spring-cloud-function-library that we recently generated under the Customer Layer drop-down.

Don't forget to select the layer's version. Because this is the first time we are deploying it, it will be version 1.

The layer will be successfully added once you press the add button. You can see if it's been added or not by looking at the layer section.

The layer's maximum zip file size is 250 MB, and you can add up to 5 layers in one AWS lambda function.(alert-passed)

Configure the handler in the AWS lambda function

To use the aws lambda function, we must first configure the request handler.

The org.springframework.cloud.function.adapter.aws.FunctionInvoker is one of the generic request handlers that is implemented in the AWS RequestStreamHandler.

Navigate to the runtime settings and click the edit button, as seen in the figure below.

Change the name of the handler to org.springframework.cloud.function.adapter.aws.FunctionInvoker as shown in the figure below, and then click the save button.

AWS Lambda Function Testing

Hurray! We have completed all of the prerequisites required to run the AWs lambda function. However, before connecting this AWS lambda function to the API Gateway to build a rest API, we must first ensure that the function is functioning properly.

To quickly validate the AWS lambda function, let's proceed to the test part.

The AWS lambda function offers us the ability to create an event to test our lambda function. Let's create an event with the name verify reverse string under the event JSON and send the following payload:

After inserting the payload, click the save button, followed by the test button.

The AWS Lambda function limits us to completing the execution of the entire function within 15 minutes, which implies we can only perform a function execution within 15 minutes.(alert-warning)

When you press the test button, the AWS lambda will respond as seen in the figure below.

You may debug your request trail by checking in the function's log, just like the response of the aws lambda.

Let's trigger our lambda function by creating the rest api using the API gateway

There are various ways of launching an AWS Lambda function, but for the purposes of this demonstration, we'll use the AWS API Gateway to do it.

The publishing, monitoring, and security of restful webservices are all handled by the fully managed Amazon API Gateway service.

The AWS API gateway serves as the entrance point for your apps before connecting to the business logic.

Are you delighted to use the API gateway to launch your first AWS lambda? If so, expose the rest endpoint to call the lambda function.

Navigate to your AWS lambda function and look for the add trigger button, as seen in the screenshot below.

The Add Trigger Configuration window will now appear. Select the API gateway from the Trigger Configuration drop-down menu.

Select the new API from the intent radio button, and ensure that the HTTP API type is selected in the API time section.

There are two ways to construct a restful webservice in the AWS API gateway, and they are as follows:
HTTP APIs: HTTP APIs have been designed with fewer functionalities and are less expensive than Rest APIs.
Rest APIs: In comparison to HTTP APIs, Rest APIs provide greater functionalities. If you require API keys, per-client throttling, request validation, AWS WAF integration, or private API endpoints, use REST APIs.(alert-success)

Choose open from the security drop-down because we're making this API open, which means we won't require API authentication to use it, and then click the add button to configure the API gateway trigger into the AWS lambda function.

Use the API endpoint provided by the API Gateway to test the AWS lambda function.

After clicking the add button, you can discover the API gateway that has been configured in the trigger section of the configuration.

You will also be provided with the API endpoint for executing the AWS lambda function.

Let's put this AWS lambda function to the test now, utilizing the API endpoint supplied by the API Gateway.

Copy and paste the API endpoint into Postman, then submit the json payload under the body, and we want the content to be reversed from the response.

As we can see from the response, we were able to retrieve the expected response from the AWS lambda function. It represents that we were able to successfully configure the AWS API gateway for our AWS lambda function.

How to build a spring cloud function in Spring Boot 2 and deliver it to AWS Lambda.

We've seen how to establish a spring cloud function in Spring Boot version 3, however there are some minor code changes in Spring Boot version 2 that we'll go over right now.

Instead of generating a bean in Spring Boot version 2, we must first develop a handler class that will be responsible for extending the SpringBootRequestHandler class.

 package in.learnjavaskills.springcloudfunctionawslambda.handler;

 import in.learnjavaskills.springcloudfunctionawslambda.dto.GatewayRequest;
 import in.learnjavaskills.springcloudfunctionawslambda.dto.GatewayResponse;

 import org.springframework.cloud.function.adapter.aws.SpringBootRequestHandler;

 public class RequestHandler extends SpringBootRequestHandler<GatewayRequest, GatewayResponse>
 {}

After you've successfully constructed a handler class, let's make one class that implements the Functiona interface. In this, we will override the apply method, which is the functional interface's abstract method, and then add code to reverse and string.

 package in.learnjavaskills.springcloudfunctionawslambda.handler;

 import in.learnjavaskills.springcloudfunctionawslambda.dto.GatewayRequest;
 import in.learnjavaskills.springcloudfunctionawslambda.dto.GatewayResponse;
 import org.springframework.stereotype.Service;

 import java.util.Objects;
 import java.util.function.Function;

 @Service
 public class RequestHandlerService implements Function<GatewayRequest, GatewayResponse>
 {

   @Override
   public GatewayResponse apply(GatewayRequest gatewayRequest)
   {
     if (Objects.nonNull(gatewayRequest) && Objects.nonNull(gatewayRequest.getText()))
     {
       String reverseText = new StringBuffer(gatewayRequest.getText()).reverse().toString();
       return new GatewayResponse(200, reverseText, "Success");
     }
     return new GatewayResponse(409, null, "fail");
   }
 }

That's all the code changes we need to make in Spring Boot version 2 versus Spring Boot 3. For Spring Boot Version 2, all dependencies will be the same.

Simply use the Spring Cloud version that is compatible with Spring Boot version 2.

Changes to the AWS lambda handler for Spring Boot version 2

The deployment to the AWS lambda function is identical to what we described earlier in this guide for Spring Boot version 3.

However, because we changed the code, we must configure the class's handler.

We created a handler called RequestHandler class. As a result, we have to provide the absolute path of a package of this class in the AWS Lambda Function's handler configuration.

in.learnjavaskills.springcloudfunctionawslambda.handler.RequestHandler~~(code-box)~~

I've already deployed the Spring Boot version 2 (Spring Cloud function), and you may edit the handler setting by navigating to the function's runtime settings, as seen in the image below.

Conclussion

In this course, we learned how to create a spring cloud function and run it on our local system. In Spring Boot Versions 2 and 3, we learn how to develop a spring cloud function and deploy it to AWS Lambda.

We not only built the spring cloud function, but we also saw how to deploy it into the AWS lambda function. We'll look at the layer in the AWS lambda function and how to use it.

We also built a Rest API by triggering our AWS lambda function through the API gateway.

Once you've finished your lab exercise, remember to delete your AWS lambda function, layer, and API Gateway.(alert-error)

Keep learning and keep growing.

How to use SpEL (Spring Expression Language) in the Spring Data JPA

2024-03-03T15:14:00.000+05:30

An expression language is used to query and operate on the objects at runtime. In Java, several expression languages are available to use, such as JBoss EL, MVFLEX Expression Language (MVEL),Object-Graph Navigation Language (ONGL) etc.

In this article, we will have some hands-on SpEL (Spring Expression Language), and I will walk you through how to use SpEL in the Spring Data JPA @Query annotation.

~~toc~~

SpEL in the Spring Data JPA

The Spring Data Jpa enables us to manually bind the parameters using the @Query annotation and this @Query annotation is also capable of binding the dynamic parameters using the SpEL (Spring Express Language).

There are a couple of approaches to hook the SpEL parameter in the Spring Data JPA repository methods. It can be an index base entry or a named parameter utilizing the @Param annotation.

Index approach

In this approach, we hook the parameters using the indexes of the method argument. The following is a simple example of SpEL in the Spring data Jpa used as the index of the method argument.

 @Query("FROM BookEntity WHERE uniqueCode = ?#{[0]} AND title = ?#{[1]}")
 List<BookEntity> findBookByUniqueBookCode(String uniqueCode, String title);

Using the named parameter in the @Param annotation

The SpEL(Spring Expression Language) can be binded using the @Param annotation to the @Query. The @Param annotation expects the value argument. The SpEL declared using the @Param annotation is shown below.

 @Query("FROM BookEntity WHERE uniqueCode = ?#{#uniqueCode} AND title = ?#{#title}")
 List<BookEntity> findBookByUniqueBookCode1(@Param("uniqueCode") String uniqueCode, @Param("title") String title);

Various patterns of initiating SpEL (Spring Expression Language)

The Spring Data JPA provides two patterns to bind the SpEL parameters in the @Query annotation. The ?# or :# can be used to activate the SpEL binding.

We looked at how to bind the SpEL parameters using the ?#. I prefer the second syntax, which is :#. In the subsequent examples, I will lead you how to use :#.

The SpEL (Spring Expression Language) utilizing the entity class.

Are you wondering if we can use named parmaters using the @param annotion without using SpEL and this will work perfectly fine? Then why should I use SpEL?

Correct. The real joy of using SpEL is with the entity class. Allow me to present to you how we can take advantage of the SpEL using the entity classes. The below snippet utilises the enity in the method argument.

 @Query("SELECT price FROM BookEntity where uniqueCode = :#{#bookEntity.uniqueCode}")
 List<Double> findBookPriceByUniqueCode(@Param("bookEntity") BookEntity bookEntity);

I have set only the uniqueCode parameter in the BookEnity because I only require the uniqueCode parameter in the query. You can configure the entity parameters to meet your conditions.

 BookEntity bookEntity = BookEntity.builder()
	.uniqueCode("BK0001")
	.build(); 
 List bookPriceByUniqueCode = bookRepository.findBookPriceByUniqueCode(bookEntity);

The SpEL (Spring Expression Language) with the relationship entity

We frequently work with relational databases, and we may be required to add entities within entities in order to map the associations in Hibernate.

If you are wondering, can we take benefit of SpEL while using the relationship hibernate mapping? The response will be yes, we can take total advantage of the SpEL in the hibernate mapping entity as well.

 @Query("FROM BookEntity WHERE authorEntity = :#{#bookEntity.authorEntity}")
 List<BookEntity> findBooksByAuthor(@Param("bookEntity") BookEntity bookEntity);

 AuthorEntity authorEntity = AuthorEntity.builder()
	.authorId(1)
	.build();
 BookEntity bookEntity = BookEntity.builder()
	.authorEntity(authorEntity)
	.build(); 
 List findBooksByAuthor = bookRepository.findBooksByAuthor(bookEntity);

Summary

In this tutorial, we learned about expression language and how to utilize SpEL (Spring Express Language) in the Spring Data JPA @Query. and We saw various patterns for binding the parameters in the SpEL and multiple ways to access them in the Spring Data JPA.

If you are curious to read the source code, you can find it here.

Keep learning and keep growing.

(getButton) #text=(Next: Configure Multiple Databases in Spring Data JPA) #icon=(link) #color=(#2339bd)

Spring Data JPA DTO Projections | Learn Java Skills

2024-03-03T15:13:00.004+05:30

The term projection means wrapping/projecting the data into the object. There are three types of widely projection used in the Spring Data JPA.

In the Spring Data JPA bydefault scaler projection is enabled. In this tutorial, we will learn how to use the DTO projection in the Spring Boot application using the Spring Data JPA.

1. Scaler projection
2. Interface projection
3. DTO projection

~~toc~~

DTO projection in the entity

We can use DTO projection on an entity to map the selected parameter to the entity class in the spring data JPA.

Let's suppose we have a table called books with the following columns, and we want to select only selected columns such as title, price, and unit_sold.

In such scenario, we can use the DTO projection on an entity to map the selected paramter into the entity; although you can use the findAll() method of the JPARepository interface, which maps the column into the entity, that could not be the best practice if you are required to select only the selected column.

On the following table, We will use the DTO projection.

 create table books(
   id bigint(20) not null auto_increment primary key,
   title varchar(255) not null,
   price float(3,2) not null,
   unit_sold bigint(200),
   author_id bigint not null,
   FOREIGN KEY (author_id) REFERENCES author(id)
 );

First Create the Entity

 public class Books
 {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private long id;
    private String title;
    private BigDecimal price;
    private long unitSold;
    @ManyToOne()
    @JoinColumn(name = "author_id", referencedColumnName = "id")
    private Author author;
	//Getter & Setter
 }

Constructor using parameters

Once we have completed creating the entity, we need to create the constructor with only the required parameters that we wish to fetch from the table. In our case, we like to fetch the title, price, and unit_sold.

In the Books Entity, Create the constructor using the desire parameter just like in the below code.

 public Books(String title, BigDecimal price, long unitSold)
 {
    this.title = title;
    this.price = price;
    this.unitSold = unitSold;
 }

JPA Query for Entity DTO projection

Let's create a JPA select query that will map the table output into the Entity. We will be using the Books constructor in the JPA query, which we created in the earlier step.

 @Query("select new Books(title, price, unitSold) from Books")
 List<Books> findAllBooks();

You can observe that new Books(title, price, unitSold) is nothing but the constructor that we created in the previous step. This constructor will map the select query output from the table into the entity class.

Test Entity DTO Projection

 @Test
 void findAllBooksTest() 
 {
   List<Books> allBooks = booksRepository.findAllBooks();
   System.out.println(allBooks);
 }

Hibernate will execute the below query internally to replace the DTO projection JPA query with the SQL query.

 Hibernate: 
    select
        b1_0.title,
        b1_0.price,
        b1_0.unit_sold 
    from
        books b1_0

Below is the output of the findAllBooks() method. As you can see, we found values of the title, price, and unit_sold columns only, and id and author_is were null because we didn't fetch those parameters in our DTO projection query in the repository.

 Books list are : 
[Books{id=0, title='Harry Potter', price=1.1, unitSold=2000, author=null}]

DTO Projection on join query

Till now, we have learnt how to use DTO projection in the one entity itself, Let's now take a glimpse how we can leverage the DTO projection technique in the join query as well.

Let's create two tables such as books and the author, and we will use an inner join query to fetch some selected parameter and will map those parameters into the DTO.

 create table books(
   id bigint(20) not null auto_increment primary key,
   title varchar(255) not null,
   price float(3,2) not null,
   unit_sold bigint(200),
   author_id bigint not null,
   FOREIGN KEY (author_id) REFERENCES author(id)
 );

 create table author(
   id bigint not null auto_increment primary key,
   name varchar(255) not null
 );

Let's create one DTO with the elements which we desire to fetch with the constructor.

 public class BooksWithAuthorDTO
 {
    private String bookTitle;
    private long bookPrice;
    private BigDecimal bookUnitSold;
    private String authorName;
	//Getter and Setter
    
    public BooksWithAuthorDTO(String bookTitle, long bookPrice, BigDecimal bookUnitSold, String authorName)
    {
        this.bookTitle = bookTitle;
        this.bookPrice = bookPrice;
        this.bookUnitSold = bookUnitSold;
        this.authorName = authorName;
    }
 }

Before we begin, First we need to learn about two annotations which are @NamedNativeQuery and @SqlResultSetMapping

@NamedNativeQuery annotation

The @NamedNativeQuery annotation is the entity level annotation and its use to create the name SQL query in the JPQL and as well as the native query.

The name should be the unique across the persistence unit because this name will be used in the JPA repository method to fetch the data.

 @NamedNativeQueries(
  @NamedNativeQuery(name = "Books.findBooksWithAuthor",
    query = "select b.title as bookTitle, b.price as bookPrice, " +
      "b.unit_sold as bookUnitSold, a.name as authorName " +
      "from books b left join author a on b.author_id = a.id;",
    resultSetMapping = "Mapping.BooksWithAuthorMapping"
  )
)

Make sure you use an alias name the same as the DTO class parameter name.

@SqlResultSetMapping annotation

The @SqlResultSetMapping annotation is also the entity level annotation. As the name of this annotation, this annotation is used to map the SQL query output into the DTO class using the constructs and instruction the parameter type to parse the correct data type into the DTO.

 @SqlResultSetMapping(name = "Mapping.BooksWithAuthorMapping",
  classes = @ConstructorResult(targetClass = BooksWithAuthorDTO.class,
    columns = {
      @ColumnResult(name = "bookTitle"),
      @ColumnResult(name = "bookPrice", type = Long.class),
      @ColumnResult(name = "bookUnitSold", type = BigDecimal.class),
      @ColumnResult(name = "authorName")
    })
 )

@ConstructorResult annotaion

The @ConstructorResult annotation will specify the DTO class in which output of the SQL query should mapped, This annotation accepts two parameters such as targetClass and columns

In the targetClass parameter we need to declare the DTO class, and under the columns we need to instruct how to use contractions to map the SQL output into the parameter using the @ColumnResult

If you have thoroughly monitored the @ColumnResult, It's anticipate the name and the type parameter, in the name we have to declare the DTO class parameter and under the type we need to render the data type of those parameters.

String data type is bydefualt hence, we don't need to explicitly declare that but for the other data type we required the appropriate data type to mapped to the fields.

JPA Rpository method

Now, Create a method in the Books Repository with the name which we used in the @NamedNativeQueries. In our case, it's findBooksWithAuthor().

 @Query(nativeQuery = true)
 List<BooksWithAuthorDTO> findBooksWithAuthor();

Test DTO projection with join query

 @Test
 void findBooksWithAuthorTest()
 {
    List<BooksWithAuthorDTO> booksWithAuthor = booksRepository.findBooksWithAuthor();
	System.out.println("Books with author are : ");
	System.out.println(booksWithAuthor);
 }

Hibernate will execute the below query internally to replace the DTO projection JPA query with the SQL query.

 Hibernate: 
    select
        b.title as bookTitle,
        b.price as bookPrice,
        b.unit_sold as bookUnitSold,
        a.name as authorName 
    from
        books b 
    left join
        author a 
            on b.author_id = a.id;

Output of the above query which has been mapped to the DTO.

 Books with author are : 
[BooksWithAuthorDTO{bookTitle='Harry Potter', bookPrice=1, bookUnitSold=2000, authorName='J.K. Rowling'}]

Conclusion

Using the DTO projection in the join query is very handy and it's leverage the advantages over the scalar projection because in the scalar projection we explicitly need to convert the Object data type into the airport data type before we need to consume it.

DTO projection enhances the code's readability and provides more flexible or easily modifiable code.

Keep learning and keep growing.

(getButton) #text=(Next: Spring Expression Language in Spring Data JPA) #icon=(link) #color=(#2339bd)

Open Feign Client | How to Consume Rest API in Spring Cloud or Spring Boot | Learn Java Skills

2024-03-03T15:13:00.003+05:30

There are many libraries available for Java to consume restful web services, and in the spring boot, Java developers mostly favor the Rest Template to consume the APIs. No doubt, the Rest Template is absolutely acceptable to use in an enterprise Java application.

The RestTemplate required us to write more code, and the more lines of code you write, the more bugs may live in your application. What if I tell you you can get rid of most of the boilerplate code of RestTemplate to consume the API?

Yes, you read correctly. We can use the Open Feign client to consume the Rest APIs in the Spring Boot applications.

If you are a Java developer and you are using the Rest Template in your Spring Boot application and you are curious to explore some more libraries to consume the Rest APIs, then you have landed in the correct article.

In this article, I'm going to walk you through how to use an open-feign client to consume the APIs in the Spring Boot application, and I can ensure that you can gain a lot of confidence from this article to use in your next production-ready code.

~~toc~~

What is Open Feign Client, and why should you use it in your next project?

Open-Feign-Client is a library to consume the Rest APIs. Open-Feign-Client comes under the umbrella of the Spring Cloud project. Some of the advantages of open feign clients are listed below.

Open Feign Cleint is widely used in the Java-based Spring Boot microservice architecture. It provides the load balancing feature out of the box by just implementing the service registry.
The URL and the port can be dynamically fetched from the service registry.
Only create abstract methods to consume the APIs.

Dependencies are required to use the Open Feign Client

Maven openfeign dependency

 <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-openfeign</artifactId>
 </dependency>

Maven Spring cloud dependency

 <dependencyManagement>
   <dependencies>
      <dependency>
	    <groupId>org.springframework.cloud</groupId>
	    <artifactId>spring-cloud-dependencies</artifactId>
	    <version>${spring-cloud.version}</version>
	    <type>pom</type>
	    <scope>import</scope>
	</dependency>
   </dependencies>
 </dependencyManagement>

You can find the lates version of above dependencies from here

Annotations

There are many annotations available for the open feign client library, but only two are widely used, and they are as follows:

@EnableFeignClients annotation

The @EnableFeignClients annotation is used at the class level of the main method. The purpose of this annotation is to instruct the spring boot application at the boot level to scan the inferfaces that are declared with the @FeignClient annotation.

 @SpringBootApplication
 @EnableFeignClients
 public class OpenFeignClientApplication 
 {
   public static void main(String[] args)
   {
	 SpringApplication.run(OpenFeignClientApplication.class, args);
   }
 }

@FeignClient annotation

The @FeignClient annotation is an interface level to declare the Rest API client. There are two main parameters used in this @FeignClient annotation, and they are as follows:

value: The name of the service, and if the service registry is enabled in the microservice architecture, then this name will be utilized by the service registry to load balance across all available instances of the service.
(alert-passed)

url: The url parameter expects the URL of the service that you are required to fetch. It should be the base URL only, not an absolute URL, for example https://www.learnjavaskills.in/
(alert-passed)

 @FeignClient(value = "student", url="localhost:8080/student/")
 public interface StudentProxy
 {
    
 }

How to consume the open Rest API using the open feign client

Let's consume one of the open APIs using the open feign client library. I've got one service running locally on my machine, and that is called the student service.

The student service exposes one rest API that shares the student's details. Take a look at the following student controller method for more clarity.

 @GetMapping("all-students")
 public ResponseEntity<List<Student>> getAllStudents()
 {
   List<Student> allStudents = studentService.getAllStudents();
   return ResponseEntity.ok(allStudents);
 }

The expected output of the above API is as follows:

 [
  {
    id: 1,
    firstname: "John",
    lastname: "pear"
  },
  {
    id: 2,
    firstname: "Mayur",
    lastname: "Rathod"
  },
  {
    id: 3,
    firstname: "Imran",
    lastname: "Shaikh"
  }
 ]

The full rest API endpoint looks like this:

http://localhost:5000/api/student/all-students~~(code-box)~~

Let's configure the above Rest endpoint in our other service where we are expecting to consume this API

Step 1: Add the base URL of the above student rest API into the application.properties file like below. The reason for adding this in the properties file is that we will hook up this base url from the application.properties files into the @FeignClient interface, where we are configuring our proxy to consume the student service.

student.base-url = http://localhost:5000/api/student/~~(code-box)~~

Step 2: Now Create one interface with the @FeignClient annotation and annotate the main method class with @EnableFeignClients. So that, Spring Boot looks up the interfaces, which are annotated with the @FeignClient, to configure the proxy.

 @FeignClient(value = "student", url="${student.base-url}")
 public interface StudentProxy
 {

 }

As you can see, the url parameter of the @FeignClient is "${student.base-url}", It's mean the value of the key, i.e., student.base-url should be fetched from the application.properties file. In this case, the value would be http://localhost:5000/api/student/ which we configured in step 1.

Step 3: Once the above setup is successfully completed, we can look forward to the next step. Let's create an abstract method that will be used to fetch the students details.

 @GetMapping("all-students")
 public List<Student> getAllStudent();

 public record Student(Long id, String firstname, String lastname)
 {}

I've use record in this example and If you are using java 8 or 11 then you can also use POJO as well

If you have closely observed that, we declare one abstract method, i.e., getAllStudent(), with the List <Student> return type, and then we have annotated this abstract method with the @GetMapping().

The intention of this @GetMapping annotation is that we are instructing the feign client that the HTTP method of the above-mentioned proxy is GET. So behind the scenes, the feign client will make an API call with the above URL using the HTTP GET method.

Unlike the HTTP GET method, Feign Client also supports other HTTP methods such as PUT for @PutMapping, POST for @PostMapping, and DELETE for @DeleteMapping annotations, respectively.

You can also use the @RequestMapping annotation on the abstract just like we use in the Rest Controller. It works perfectly fine. You can give it a try.

It's time to test feign client API calling

Let's create a service class to call the StudentProxy interface the way we usually do.

 @org.springframework.stereotype.Service
 public class Service
 {
   private final StudentProxy studentProxy;

   @Autowired
   public Service(StudentProxy studentProxy)
   {
     this.studentProxy = studentProxy;
   }

   public List<Student> getStudents()
   {
     return studentProxy.getAllStudent();
   }
}

Now, we will call the above service method to print the data that we are getting from the Student Service Rest API. I've got to write code in the main method because I don't want to expose any rest API endpoints in that service.

 @SpringBootApplication
 @EnableFeignClients
 public class OpenFeignClientApplication
 {
   public static void main(String[] args)
   {
	 ConfigurableApplicationContext configurableApplicationContext = SpringApplication.run(OpenFeignClientApplication.class, args);
	 Service service = (Service) configurableApplicationContext.getBean("service");
	 System.out.println("Fetching all students");
	 List<Student> allStudent = service.getStudents();
	 allStudent.stream()
	   .forEach(System.out :: println);
   }
 }

The output of the above code is as follows:

Fetching all students
Student[id=1, firstname=John, lastname=pear]
Student[id=2, firstname=Mayur, lastname=Rathod]
Student[id=3, firstname=Imran, lastname=Shaikh]~~(code-box)~~

ResponseEntity in the feign client interface

If you are curious to know, before getting the response from the body, I want to first check the header of the response or the status code of the response.

The feign client also supports the ResponseEntity class. With the help of the ResponseEntity class, you can read the header, status, and body of the response. You just need to replace the return type with the ResponseEntity class instead of using the body objects, just like below.

 @FeignClient(value = "student", url="${student.base-url}")
 public interface StudentProxy
 {
   @GetMapping("all-students")
   ResponseEntity<List<Student>> getAllStudent();
 }

Service can be changed as follows:

 public List<Student> getStudents()
 {
   ResponseEntity<List<Student>> response = studentProxy.getAllStudent();
   if (response.getStatusCode().value() == 200)
     return response.getBody();
   return Collections.emptyList();
 }

How to consume a secure Rest API using an open feign client?

Till now, we have only seen how to consume the open Rest API. If you are working in an enterprise Java application, then you are most likely consuming the secure API. Let's dig into how we will be able to pass an authorization token in the header of the API request.

Yes, you read correctly. In this section, we will also learn how we are going to pass data in the API request header using the open feign client. However, I'm taking an example of the authorization; you can feel free to send any other request from the header. The process will be exactly the same.

Are you excited to explore how to send authorization tokens? Let's start right below.

What is the @RequestHeader annotation in the feign client?

The annotation request header means that data should be passed to the request header of the Rest API. As a developer, you may need to send some data in the header of the API request. In such a case, you can utilize the @RequestHeader annotation.

There are two widely used parameters in the @RequestHeader annotation, and they are as follows:

value: The value defines which header parameter should be aligned. In our case, we are going to send the Basic token under the authorization parameter of the request header for authentication and authorization. (alert-passed)

required: The required argument of the @RequestHeader annotation accepts the boolean value. This argument can be utilized to instruct whether the request header parameter is mandatory to consume the API. By default, it's true. (alert-passed)

Update the code to send the authorization token in the request header

In this example, we are going to use the same student API; I've just enabled the spring security in that service to demonstrate this exercise.

Step 1: Update the student proxy to accept the authorization token in the parameter and annotate it with the @RequestHeader, just like below.

 @FeignClient(value = "student", url="${student.base-url}")
 public interface StudentProxy
 {
   @GetMapping("all-students")
   ResponseEntity<List<Student>> getAllStudent(@RequestHeader(value = "Authorization", required = true) String   authorizationToken);
 }

Step 2: Modify the service layer to generate the Bassic authorization token and pass it to the student proxy just like below.

 public List<Student> getStudents()
 {
   String username = "learnjavaskills.in";
   String password = "password";

   String credential = username + ":" + password;
   byte[] encoded = Base64.getEncoder().encode(credential.getBytes());

   String authorizationToken = "Basic " + new String(encoded);

   ResponseEntity<List<Student>> response = studentProxy.getAllStudent(authorizationToken);
   if (response.getStatusCode().value() == 200)
     return response.getBody();
   return Collections.emptyList();
 }

In this section, we have seen how we can send the Basic token, but in most APIs, you may be required to send the Bearer token. In that case, the process is exactly the same: you can use @RequestHeader in the proxy, and you just need to change the mechanism of generating the bearer token.

Dynamic URL in the open feign client

There are some scenarios where you need to consume some of the Rest API by using the dynamic URL. Till now, we have consumed an API with a static URL. Are you excited to learn how to send the URL in the open feign client at runtime? If yes, then let's dive into it.

Before working on the runtime URL, first understand when we may encounter such a requirement. One may say that if the API uses HATEOAS (Hypermedia as the Engine of Application State) that case we can get several URLs apart from the data in the Rest API response, and we are now bound to execute those links from the HATEOAS response.

This is one of my favorite parts of this: connecting the API with the runtime URL.

How URIs are used in the open feign client

To consume the rest API by sending the runtime url in the open feign client, we just need to accept the URI parameter in the proxy abstract method, just like below: Again, we are using the same student API, but this time we send the URL from the run time.

 @FeignClient(value = "student", url="${student.base-url}")
 public interface StudentProxy
 {
   @GetMapping()
   ResponseEntity<List<Student>> getAllStudent(@RequestHeader(value = "Authorization", required = true) String authorizationToken,
            URI uri);
 }

If you have closely monitored that, this time we are not appending any static URL under the @GetMapping annotation, and still open feign clients will be able to consume APIs because of those URI parameters.

Now we need to convert the string url into the URI and pass it to the student proxy abstract method.

 public List<Student> getStudents() throws URISyntaxException
 {
   String username = "learnjavaskills.in";
   String password = "password";

   String credential = username + ":" + password;
   byte[] encoded = Base64.getEncoder().encode(credential.getBytes());

   String authorizationToken = "Basic " + new String(encoded);

   String url = "http://localhost:5000/api/student/all-students";
   URI uri = new URI(url);

   ResponseEntity<List<Student>> response = studentProxy.getAllStudent(authorizationToken, uri);
   if (response.getStatusCode().value() == 200)
     return response.getBody();
   return Collections.emptyList();
 }

How to post some data using the open feign client

Just like earlier I said we can use all the HTTp methods in the open feign client, it's time to understand if we want to send data to the API and how we will be able to achieve this in the open feign client. Let's take a look below.

Create an abstract method in the feign client interface and annotate that method with @PostMapping this time, and add parameters to send the data in the request body of the API. Here we are sending the Student object to add students.

 @FeignClient(value = "student", url="${student.base-url}")
 public interface StudentProxy
 {
   @PostMapping("add-student")
   ResponseEntity<Long> addStudent(Student student);
 }

Your service class may look like the one below to pass the student object.

 public Long addStudent()
 {
   ResponseEntity<Long> response = studentProxy.addStudent(new Student("Jack", "abram"));
   if (response.getStatusCode().value() == 200)
     return response.getBody();
   return 0L;
 }

Just like this, you can also leverage the @RequestParam and the @PathVaraible. If you want me to explain how to use these two annotations in the open feign client, then you can comment in this post, and I'll be happy to add to this.

Logging in to an open feign client request and response

Debugging is a crucial part of development. It helps us monitor the request flow of the API and find out the exact root cause of the issue. You can also dubug your API request in the open feign client.

Enabling logging from the application.properties file

Let's understand what an open-feign client offers us to debug our API request. First, we must enable debug mode in the application. properties file, following is the syntax

logging.level.<project-package> = DEBUG~~(code-box)~~

logging.level.in.learnjavaskills.openfeignclient = DEBUG~~(code-box)~~

Configure the level of the log

Once we have successfully enabled logging, we need to configure the level of the log by creating a bean, just like below.

 package in.learnjavaskills.openfeignclient;

 import feign.Logger;
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;

 @Configuration
 public class OpenFeignClientConfig
 {
   @Bean
   public Logger.Level feignLoggerLevel()
   {
     return Logger.Level.FULL;
   }
 }

There are four levels of log in the open feign client log configuration. In the above example, we have enabled FULL log. Let's dive into the deepest level of the log.

NONE: This level of log is enabled by default. As the name indicates, none means no log will be printed on the console.

BASIC: If you want to print only the basic log in the console, such as the request url, response status, and HTTP method used in the API, or you want to monitor the execution time, then the BASIC logging level is best for your case.

HEADERS: The HEADER level will enable the basic level and the request and response API header details to be printed in the console. This can be a good option to enable in production, but if you are happy with the basic details, then I highly encourage you to opt for the BASIC level only.

FULL: The final level is FULL, It will print all the request and response flows of the API. This level is best if you are in the development phase.

Let's see the console how logs are populated

 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         :  [StudentProxy#addStudent] ---> POST http://localhost:5001/api/student/add-student HTTP/1.1
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] Content-Length: 46
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] Content-Type: application/json
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] 
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] {"id":0,"firstname":"Jack","lastname":"abram"}
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] ---> END HTTP (46-byte body)
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] <--- HTTP/1.1 200 (25ms)
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] connection: keep-alive
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] content-type: application/json
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] keep-alive: timeout=60
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] transfer-encoding: chunked
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] 
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] 10
 Time stamp with date DEBUG 12116 --- [           main] i.l.openfeignclient.StudentProxy         : [StudentProxy#addStudent] <--- END HTTP (2-byte body)

Conclusion

In this tutorial, we have seen how to use the open feign client to consume the Rest API, and we have also learned how to enable the log level in the feign client to debug or monitor API requests and the response.

We have discussed in detail all the annotations that we can use in the opein feign client. I'll strongly suggest you do a hand on it to understand each and every topic in this article in detail.

Feel free to comment on this article about your thoughts or opinions; I'll be happy to address that.

Keep learning and keep growing.

Spring Data JPA Composite Primary Key | Learn Java Skills

2024-03-03T15:12:00.001+05:30

In the enterprise database architecture, it's most common to use the composite primary key. Spring Data JPA offers us two ways to map the composite primary key in the entity.

In this article, you will learn how to map the composite primary key in Spring Data JPA using the @IdClass and @EmbeddedId annotations, as well as the @Embeddable annotation.

~~toc~~

What is a composite primary key?

Before we get into the composite primary key, let's first define the primary key.

In SQL, a primary key is a constraint that is unique and not null across table rows.

Similarly, the composite key is a primary key, but the distinction between the two is that the composite key is the combination of more than one primary key in SQL to identify the unique row in the table.

Overview of Books Table

Before we go into the design of the composite key, let's first go over the table structure of the books.

In order to demonstrate this lecture, I've created a books table with two primary keys, namely book_id and title, also known as the composite key in a table.

I have extra fields in addition to the composite key. You may construct the table on your local system by running the SQL script below, or you can use your existing table if you have one.

 create database composite_key_tutorial;

 use composite_key_tutorial;

 create table books(
    book_id BIGINT AUTO_INCREMENT,
    title VARCHAR(50),
    book_description VARCHAR(255),
    price DOUBLE,
    unit_sold BIGINT default 0,
    available_quantity BIGINT default 0, 
    PRIMARY KEY(book_id, title)
 );

Composite Key using @IdClass Annotation

Let's first use the @IdClass annotation to begin mapping the composite key in the sprind data JPA.

The jakarta.persistence package contains the @IdClass annotation. The @IdClass annotation defines a composite primary key class that is assigned to numerous entity fields or attributes.

Composite Primary Key Class

If we use the @IdClass annotation in the Spring Data JPA to map the composite key, we must build one POJO for the primary key fields.

I'm going to create a BooksId class for this lecture. Remember that the books table has two primary keys: book_id and title. In this BooksId class, we will include both of these primary key fields.

 package in.learnjavaskills.compositekeytutorial.entity;

 import java.io.Serializable;
 import java.util.Objects;

 public class BooksId implements Serializable
 {
    private Long bookId;
    private String title;

    public BooksId()
    {
    }

    public BooksId(Long bookId, String title)
    {
        this.bookId = bookId;
        this.title = title;
    }

    @Override
    public boolean equals(Object o)
    {
        if (this == o)
            return true;
        if (o == null || getClass() != o.getClass())
            return false;
        BooksId booksId = (BooksId) o;
        return Objects.equals(bookId, booksId.bookId) && Objects.equals(title, booksId.title);
    }

    @Override
    public int hashCode()
    {
        return Objects.hash(bookId, title);
    }
 }

The composite primary key class must implement the Serializable interface, have no and all-argument constructors, and define the equal and hashcode methods.(alert-passed)

Books Enity Class

After successfully creating the composite primary key class, ensure that you declare this ID class in your entity class using the @IdClass annotation, as shown below.

 package in.learnjavaskills.compositekeytutorial.entity;

 import jakarta.persistence.Entity;
 import jakarta.persistence.Id;
 import jakarta.persistence.IdClass;
 import jakarta.persistence.Table;

 import java.math.BigDecimal;
 import java.util.Objects;

 @Entity
 @Table(name = "books")
 @IdClass(BooksId.class)
 public class BooksEntity
 {
    @Id
    private Long bookId;

    @Id
    private String title;

    private String bookDescription;

    private BigDecimal price;

    private Long unitSold;

    private Long availableQuantity;

    // getter, setter, toString, hashcode, equal
 }

That's all there is to it for mapping the composite primary key in Spring Data JPA using the @IdClass annotation.

Books Repository Interface

I've also made a repository to test the mapping, which is seen below. I'll run some database transactions to double-check the mapping.

 package in.learnjavaskills.compositekeytutorial.repository;

 import in.learnjavaskills.compositekeytutorial.entity.BooksEntity;
 import in.learnjavaskills.compositekeytutorial.entity.BooksId;
 import org.springframework.data.jpa.repository.JpaRepository;
 import org.springframework.stereotype.Repository;

 @Repository
 public interface BooksRepository extends JpaRepository<BooksEntity, BooksId>
 {

 }

As seen in the above code, the JpaRepository interface must define generic, i.e., entity, and primary key data type.

Typically, we declare the primary key's data type. What if we use the composite primary key? In this case, we may utilize the BooksId composite key class that we generated before.

It's time to put the code to the test.

I'll use the JUnit and Mockito libraries to unit test the code. In the following test, I attempted to get all of the data from the database and insert some books into the table to confirm that the configuration was functioning properly.

 package in.learnjavaskills.compositekeytutorial.repository;

 import in.learnjavaskills.compositekeytutorial.entity.BooksEntity;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase;
 import org.springframework.boot.test.autoconfigure.orm.jpa.DataJpaTest;

 import java.math.BigDecimal;
 import java.util.List;

 @DataJpaTest
 @AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
 class BooksRepositoryTest
 {
    @Autowired
    private BooksRepository booksRepository;

    @Test
    void testFindAll()
    {
        List<BooksEntity> booksRepositoryAll = booksRepository.findAll();
        Assertions.assertThat(booksRepositoryAll).isNotNull()
                .isNotEmpty();
    }

    @Test
    void saveBook()
    {
        BooksEntity booksEntity = new BooksEntity(); 
        booksEntity.setBookId(4L);
        booksEntity.setTitle("Clean Code");
        booksEntity.setBookDescription("Best book to learn writing clean code");
        booksEntity.setPrice(BigDecimal.TEN);
        booksEntity.setUnitSold(0L);
        booksEntity.setAvailableQuantity(100L);

        BooksEntity save = booksRepository.save(booksEntity);
        Assertions.assertThat(save).isNotNull();
    }
 }

Composite Key using @EmbeddedId and @Embeddable Annotation

It's time to learn about the second approach for mapping the composite primary key in Spring Data JPA, which involves using the @Embeddable and the @EmbeddedId annoations.

This approach is commonly used in corporate applications. To simplify things, I'm going to utilize the same books table from the previous example to demonstrate this approach.

@Embeddable	@EmbeddedId
The `@Embeddable` annotation denotes a class whose instances are kept as an inherent element of an owner entity and share its identity. Each of the embedded object's permanent characteristics or fields is mapped to the entity's database table.	`@EmbeddedId` is a unique identifier. To describe a composite primary key that is an embeddable class, this term is used to a persistent field or attribute of an entity class or mapped superclass. When the EmbeddedId annotation is used, there must be only one EmbeddedId annotation and no Id annotation
The `jakarta.persistence` package has both @EmbeddedId and @Embeddable.

Composite Primary Key Class with @Embeddable annotation

To include into the entity class, let's construct an Embeddable composite primary key class. Add each of the table's primary keys to this class.

 package in.learnjavaskills.compositekeytutorial.entity;

 import jakarta.persistence.Embeddable;

 import java.io.Serializable;
 import java.util.Objects;

 @Embeddable
 public class BooksId implements Serializable
 {
    private Long bookId;
    private String title;

    public BooksId()
    {
    }

    public BooksId(Long bookId, String title)
    {
        this.bookId = bookId;
        this.title = title;
    }

    @Override
    public boolean equals(Object o)
    {
        if (this == o)
            return true;
        if (o == null || getClass() != o.getClass())
            return false;
        BooksId booksId = (BooksId) o;
        return Objects.equals(bookId, booksId.bookId) && Objects.equals(title, booksId.title);
    }

    @Override
    public int hashCode()
    {
        return Objects.hash(bookId, title);
    }

    @Override
    public String toString()
    {
        return "BooksId{" + "bookId=" + bookId + ", title='" + title + '\'' + '}';
    }
 }

In addition to implementing the Serializable Interface and defining the equal and hashcode methods, the Composite primary key class needs to be annotated with the @Embeddable annotations.(alert-passed)

Books Enity Class with @EmbeddedId annotation

To map it, embed the composite primary key class into the entity class as shown below.

 package in.learnjavaskills.compositekeytutorial.entity;

 import jakarta.persistence.*;

 import java.math.BigDecimal;
 import java.util.Objects;

 @Entity
 @Table(name = "books")
 public class BooksEntity
 {
    @EmbeddedId
    private BooksId bookId;

    private String bookDescription;

    private BigDecimal price;

    private Long unitSold;

    private Long availableQuantity;
    
    // getter, setter, hashcode, equal method
 }

It's time to put the code to the test.

Because the BooksRepository interface remains unchanged, I have not included it in the following manner to reduce repetition. Let's run the code through unit testing to see whether the setup works as expected.

 package in.learnjavaskills.compositekeytutorial.repository;

 import in.learnjavaskills.compositekeytutorial.entity.BooksEntity;
 import in.learnjavaskills.compositekeytutorial.entity.BooksId;
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.Test;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.test.autoconfigure.jdbc.AutoConfigureTestDatabase;
 import org.springframework.boot.test.autoconfigure.orm.jpa.DataJpaTest;

 import java.math.BigDecimal;
 import java.util.List;

 @DataJpaTest
 @AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
 class BooksRepositoryTest
 {
    @Autowired
    private BooksRepository booksRepository;

    @Test
    void testFindAll()
    {
        List<BooksEntity> booksRepositoryAll = booksRepository.findAll();
        Assertions.assertThat(booksRepositoryAll).isNotNull()
                .isNotEmpty();

        booksRepositoryAll.forEach(System.out :: println);
    }

    @Test
    void saveBook()
    {
        BooksEntity booksEntity = new BooksEntity();
        BooksId booksId = new BooksId(4L, "Clean Code");

        booksEntity.setBookId(booksId);
        booksEntity.setBookDescription("Best book to learn writing clean code");
        booksEntity.setPrice(BigDecimal.TEN);
        booksEntity.setUnitSold(0L);
        booksEntity.setAvailableQuantity(100L);

        BooksEntity save = booksRepository.save(booksEntity);
        Assertions.assertThat(save).isNotNull();
    }
 }

The distinctions between @IdClass and @EmbeddedId

	@IdClass	@EmbeddedId
Primary Key Redundancy	The primary key must be specified twice in the `@IdClass`, once in the composite primary key class and once in the enity class.	The `@EmbeddedId` annotation is more robust than the @IdClass annotation since it allows us to bundle the composite primary key in one class, giving you a clear representation of the composite key fields because they are all aggregated in one single class that is only available through the enity class.
Custom JPQL Query	It is simple to create a custom JPQL query for the primary key. `SELECT b.bookId, b.title FROM Books b;`	If you want to query the composite key, you may need to create a JPQL query like this. `SELECT b.booksId.bookId, b.booksId.title FROM Books b;`

Conclussion

We covered how to map the composite primary key in Spring Data JPA in this lecture. We study the two approaches for mapping the composite key in Spring Data JPA: @IdClass and @EmbeddedId.

If you don't mind putting more text on the custom JPQL query, don't want the redundancy of defining an ID twice, and want to bundle a composite key in one class, use @EmbeddedId.

Keep learning and keep growing.

Spring Cloud Stream and Kafka Streams: The Perfect Duo for Building Event-Driven Architectures | LearnJavaSkills

2024-03-03T15:12:00.000+05:30

Event-Driven Architecture is a software design pattern or model to build an application. The even-driven systems are designed to capture and process the event between the decouple services.

In an even-driven architecture, two systems can communicate with each other asynchronously.

The source code is available on (getButton) #text=(GitHub) #icon=(download) #color=(#000000)

~~toc~~

What is Spring Cloud Stream?

The Spring Cloud Stream is a framework that comes under the umbrella of the Spring Cloud. The Spring Cloud stream offers us the opportunity to build an event-driven microservice architecture.

The framework offers a versatile programming architecture with support for persistent pub/sub semantics, consumer groups, and stateful partitions, all of which are built on well-known and established Spring idioms and best practices.

Spring Cloud Stream has a Binder abstraction that may be used to connect to physical destinations at external middleware.

Spring Cloud Stream supports a number of binder implementations, including but not limited to RabbitMQ, Apache Kafka, and Amazon Kinesis. In this lecture, we will use the Kafka stream.

What is Apche Kafka?

Apache Kafka is a distributed event streaming platform that is open source and used by hundreds of businesses for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

It is a publish-subscribe messaging system that functions as an interface for two parties: the sender and the recipient. More information on kafka may be found getButton) #text=(Apache Kafka: The Secret Sauce to Real-Time Data Streaming) #icon=(link) #color=(#2339bd)

Overview of the Spring Cloud Stream and Kafka Stream projects

In this lecture, we will build a project named BookMyMovie. This project will be in charge of booking the movie show, and we will construct this software leveraging the even-driven architecture with the spring cloud stream and kafka stream.

To begin, we will construct the Rest API by utilizing the Spring cloud stream and the Kafka stream to receive the payload (a movie ticket book request), which will then send the payload data in JSON format to the Kafka topic, i.e., movie-ticket-request-topic.

We will create a stream processing with Kafka stream in order to look for the event in the movie-ticket-request-topic kafka topic and perform real-time activities before sending to the movie-ticket-booked-topic kafka topic.

After that, we'll look at how to consume messages from the movie-ticket-request-topic kafka topic.

BookMyMovie Service

The application design is as follows: a user sends a request to the backend Rest API, after which the application validates the movie ticket request and transforms the data before sending it back to the Kafk topic i.e., movie-ticket-request-topic.

Dependencies

In your project, include the following dependencies: The build.gradle file is shown below.

 plugins {
	id 'java'
	id 'org.springframework.boot' version '3.1.6'
	id 'io.spring.dependency-management' version '1.1.4'
 }

 group = 'in.learnjavaskills'
 version = '0.0.1-SNAPSHOT'

 java {
	sourceCompatibility = '17'
 }

 repositories {
	mavenCentral()
 }

 ext {
	set('springCloudVersion', "2022.0.4")
 }

 dependencies {
	implementation 'org.springframework.boot:spring-boot-starter-web'
	implementation 'org.springframework.cloud:spring-cloud-stream'
	implementation 'org.springframework.cloud:spring-cloud-stream-binder-kafka'
	implementation 'org.springframework.kafka:spring-kafka'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'org.springframework.cloud:spring-cloud-stream-test-binder'
	testImplementation 'org.springframework.kafka:spring-kafka-test'
 }

 dependencyManagement {
	imports {
		mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
	}
 }

 tasks.named('bootBuildImage') {
	builder = 'paketobuildpacks/builder-jammy-base:latest'
 }

 tasks.named('test') {
	useJUnitPlatform()
 }

Configuration of the Kafka Binder Properties

The spring.cloud.stream.kafka.binder.brokers property is the list of brokers to which the Kafka binder connects.

Default: localhost(alert-success)

 spring.cloud.stream.kafka.binder.brokers = 3.80.128.59:9092

Because I'm hosting my Kafka server on an AWS EC2 instance in a single broker node, I must supply my EC2 instance's public IP address together with the right port, which in my case is 3.80.128.59:9092.

As you can see here, I utilized the IP address of the Kafak broker server node, however this is not the best practice. You should always utilize host names rather than IP addresses.(alert-warning)

If you are running Kafka on your local system, you do not need to include this setting in your application. Spring cloud stream search kafa broker server is on localhost:9092 by default.

Controller

To accept user input, we may offer a single rest endpoint that will receive user input and then transmit the same data to the kafka topic, i.e., movie-ticket-request-topic.

 package in.learnjavaskills.bookmymovieservice.controller;

 import in.learnjavaskills.bookmymovieservice.dto.MovieTicketBookingRequest;
 import in.learnjavaskills.bookmymovieservice.service.InitiateMovieTicketBookingService;
 import org.springframework.http.ResponseEntity;
 import org.springframework.web.bind.annotation.PostMapping;
 import org.springframework.web.bind.annotation.RequestBody;
 import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.RestController;

 @RestController
 @RequestMapping("/movies/")
 public class BookMyMovieController
 {
    private final InitiateMovieTicketBookingService initiateMovieTicketBookingService;

    public BookMyMovieController(InitiateMovieTicketBookingService initiateMovieTicketBookingService) {
        this.initiateMovieTicketBookingService = initiateMovieTicketBookingService;
    }

    @PostMapping("book-ticket")
    ResponseEntity<String> bookMovieTicket(@RequestBody MovieTicketBookingRequest movieTicketBookingRequest) {
        return initiateMovieTicketBookingService.publishMovieTicketBookingMessage(movieTicketBookingRequest);
    }
 }

The MovieTicketBookingRequest.class, as shown in the book-ticket API, is used to accept the user's request payload. Let's make the MovieTicketBookingRequest class quickly.

 package in.learnjavaskills.bookmymovieservice.dto;

 import com.fasterxml.jackson.annotation.JsonFormat;
 import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
 import com.fasterxml.jackson.databind.annotation.JsonSerialize;
 import com.fasterxml.jackson.datatype.jsr310.deser.LocalDateTimeDeserializer;
 import com.fasterxml.jackson.datatype.jsr310.ser.LocalDateTimeSerializer;

 import java.time.LocalDateTime;
 import java.util.Objects;

 public class MovieTicketBookingRequest
 {
    private String movieName;
    private byte screenNumber;
    @JsonDeserialize(using = LocalDateTimeDeserializer.class)
    @JsonSerialize(using = LocalDateTimeSerializer.class)
    @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
    private LocalDateTime movieShowTime;
    private String seatNumber;
    private String user;

   // getter, setter, toString, equal and hashcode method's
 }

InitiateMovieTicketBookingService class

It's time to build some business logic in InitiateMovieTicketBookingService.class and then send a message to the movie-ticket-request-topic Kafka topic.

 package in.learnjavaskills.bookmymovieservice.service;

 import in.learnjavaskills.bookmymovieservice.dto.MovieTicketBookingMessage;
 import in.learnjavaskills.bookmymovieservice.dto.MovieTicketBookingRequest;
 import org.springframework.cloud.stream.function.StreamBridge;
 import org.springframework.http.HttpStatus;
 import org.springframework.http.ResponseEntity;
 import org.springframework.messaging.Message;
 import org.springframework.messaging.support.MessageBuilder;
 import org.springframework.stereotype.Service;
 import org.springframework.util.MimeTypeUtils;

 import java.util.Objects;

 @Service
 public class InitiateMovieTicketBookingService
 {

    private final StreamBridge streamBridge;

    private final String topicName = "movie-ticket-request-topic";

    public InitiateMovieTicketBookingService(StreamBridge streamBridge) {
        this.streamBridge = streamBridge;
    }

    /**
     * Publishing message to the kafka topic
     * @param movieTicketBookingRequest
     * @return response
     */
    public ResponseEntity<String> publishMovieTicketBookingMessage(MovieTicketBookingRequest movieTicketBookingRequest) {
        if (Objects.isNull(movieTicketBookingRequest))
            return ResponseEntity.badRequest().body("Movie Ticket Booking Request must be non null");

        MovieTicketBookingMessage movieTicketBookingMessage = new MovieTicketBookingMessage(movieTicketBookingRequest);
        Message<MovieTicketBookingMessage> movieTicketBookingMessageMessage = generateMessage(movieTicketBookingMessage);
        boolean isMessageSend = streamBridge.send(topicName, movieTicketBookingMessageMessage, MimeTypeUtils.APPLICATION_JSON);
        if (isMessageSend)
            return ResponseEntity.ok("Movie booking ticket initiated success");
        else return ResponseEntity.status(HttpStatus.EXPECTATION_FAILED)
                .body("Unable to send message to kafka topic");
    }


    /**
     * Generate message to send to the kafka topic
     * @param movieTicketBookingMessage
     * @return Message
     */
    private Message<MovieTicketBookingMessage> generateMessage(MovieTicketBookingMessage movieTicketBookingMessage) {
        if (Objects.isNull(movieTicketBookingMessage))
            return null;

        return MessageBuilder.withPayload(movieTicketBookingMessage)
                .build();
    }
 }

Let's build a MovieTicketBookingMessage record that is in charge of collecting Kafka message data.

 package in.learnjavaskills.bookmymovieservice.dto;

 import com.fasterxml.jackson.annotation.JsonFormat;

 import java.time.LocalDateTime;

 public record MovieTicketBookingMessage(String movieName, byte screenNumber,
         @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
         LocalDateTime movieShowTime,
         String seatNumber,
         @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
         LocalDateTime messageGeneratedOn,
         String users)
 {
    public MovieTicketBookingMessage(MovieTicketBookingRequest movieTicketBookingRequest) {
        this (movieTicketBookingRequest.getMovieName(),
                movieTicketBookingRequest.getScreenNumber(),
                movieTicketBookingRequest.getMovieShowTime(),
                movieTicketBookingRequest.getSeatNumber(),
                LocalDateTime.now(),
                movieTicketBookingRequest.getUser());
    }
 }

In the spring cloud stream, how do I send a message to the Kafka topic?

To deliver the message into the Kafka topic, the Spring cloud stream provides the StreamBridge class. The StreamBridge class is part of the org.springframework.cloud.stream.function package.

The StreamBridge class allows users to transfer data to an output binding.

While the user rarely needs to manually transmit data in a normal spring-cloud-stream application situation, there are instances when the sources of data are outside of the spring-cloud-stream context, and so we need to bridge such foreign sources with spring-cloud-stream.

Because our data comes from an API, we must utilize StreamBridge in this scenario.

We can transmit the message to the Kafka topic using the send method of the StreamBridge class. The StreamBridge class contains an extensive number of overloaded send methods. Here in the above-mentioned example, we utilize the following method:.

 public boolean send(String bindingName, Object data, MimeType outputContentType)

The bindingName is simply the name of the Kafka topic. The data argument can represent our kafak message, and the MimeType field can describe what sort of message we are delivering to the kafka topic.

Message interface in the spring cloud stream

The Kafa message payload can be wrapped in the Message interface. The Message interface is found in the package org.springframework.messaging and is used for representing messages with the header and body.

The generateMessage() method wraps the payload, i.e., MovieTicketBookingMessage, inside the Message interface.

If you're wondering how I can send the message to the Kafka topic with the key, that's also feasible using the Message interface since we simply need to include the header for the key to send the message into the Kafka topic with the key.

 MessageBuilder.withPayload(movieTicketBookingMessage)
     .setHeader(KafkaHeaders.KEY, "YOUR_KEY_HERE".getBytes())
     .build();

Booking Process Service

Create a streaming application with Spring Cloud Stream and Kafka Stream.

The booking-process-service is a streaming service that reads the Kafka message from the movie-ticket-request-topic and writes it to the movie-ticked-booked-topic.

The Kafka Streams binder supports the three primary forms of Kafka Streams: KStream , KTable , and GlobalKTable .

Functional Style

Beginning with Spring Cloud Stream 3.0.0, the Kafka Streams binder allows applications to be created and developed utilizing the functional programming style accessible in Java 8.

This means that the applications may be expressed concisely as a lambda expression of the kinds java.util.function.Function or java.util.function.Consumer .

Process Movie Ticket Booking Function

Let's use the functional interface to develop a function(processMovieTicketBooking) for Kafka streaming. This function is in charge of reading the message from the movie-ticket-request-topic and writing it back into the movie-ticked-booked-topic.

 package in.learnjavaskills.bookingprocessservice.function;

 import in.learnjavaskills.bookingprocessservice.dto.MovieTicketBookedSuccessfully;
 import in.learnjavaskills.bookingprocessservice.dto.MovieTicketBookingMessage;
 import org.apache.kafka.streams.KeyValue;
 import org.apache.kafka.streams.kstream.KStream;
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;

 import java.nio.charset.StandardCharsets;
 import java.security.MessageDigest;
 import java.security.NoSuchAlgorithmException;
 import java.time.LocalDateTime;
 import java.util.Base64;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.UUID;
 import java.util.function.Function;

 @Configuration
 public class MovieTicketBookingProcess
 {

    @Bean("processMovieTicketBooking")
    public Function<KStream<String, MovieTicketBookingMessage>, KStream<String, MovieTicketBookedSuccessfully>> processMovieTicketBooking() {
        return movieBookingRequestMessageKStream -> movieBookingRequestMessageKStream.map( (messageKey, movieBookingRequestMessage) -> {
            System.out.println("New message arrived with key : " + messageKey + " and values : " + movieBookingRequestMessage.toString());
            Optional<MovieTicketBookedSuccessfully> movieTicketBookedSuccessfully = bookMovieTicket(movieBookingRequestMessage);
            return new KeyValue<String, Optional<MovieTicketBookedSuccessfully>>(messageKey, movieTicketBookedSuccessfully);
        }).filter((messageKey, movieTicketBookedSuccessfullyOptional) -> {
            if (movieTicketBookedSuccessfullyOptional.isPresent()) {
                MovieTicketBookedSuccessfully movieTicketBookedSuccessfully = movieTicketBookedSuccessfullyOptional.get();
                System.out.println("Booking movie ticket for movie id : " + movieTicketBookedSuccessfully.movieId());
                return  true;
            } return false;
        }).map((messageKey, movieTicketBookedSuccessfully) -> new KeyValue<>(messageKey, movieTicketBookedSuccessfully.get()));
    }

    private Optional<MovieTicketBookedSuccessfully> bookMovieTicket(MovieTicketBookingMessage movieTicketBookingMessage) {
        if (Objects.isNull(movieTicketBookingMessage))
            return Optional.empty();

        Optional<String> moviePassCode = generateMoviePassCode(movieTicketBookingMessage);
        if (!moviePassCode.isPresent())
            return Optional.empty();

        MovieTicketBookedSuccessfully movieTicketBookedSuccessfully = new MovieTicketBookedSuccessfully(movieTicketBookingMessage.movieName() + UUID.randomUUID().toString(),
                movieTicketBookingMessage.users(),
                moviePassCode.get(),
                LocalDateTime.now());
        return Optional.of(movieTicketBookedSuccessfully);
    }

    private Optional<String> generateMoviePassCode(MovieTicketBookingMessage movieTicketBookingMessage) {
        if (Objects.isNull(movieTicketBookingMessage))
            return Optional.empty();

        String code = movieTicketBookingMessage.movieName() + " " + movieTicketBookingMessage.users()
                      + " " + movieTicketBookingMessage.seatNumber();

        try  {
            MessageDigest messageDigest = MessageDigest.getInstance("SHA-256");
            byte[] digest = messageDigest.digest(code.getBytes(StandardCharsets.UTF_8));
            return Optional.of(Base64.getEncoder().encodeToString(digest));
        }
        catch (NoSuchAlgorithmException e) {
            return Optional.empty();
        }
    }
 }

For the sake of simplicity, I've omitted the database communication here. You may also utilize the database to validate or store movie tickets details. (alert-passed)

The processMovieTicketBooking function is binding with the single inbound with the KStream<String, MovieTicketBookingMessage> and single outbound with the KStream<String, MovieTicketBookedSuccessfully>.

 package in.learnjavaskills.bookingprocessservice.dto;

 import com.fasterxml.jackson.annotation.JsonFormat;
 import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
 import com.fasterxml.jackson.datatype.jsr310.deser.LocalDateTimeDeserializer;

 import java.time.LocalDateTime;

 public record MovieTicketBookingMessage(String movieName, byte screenNumber,
         @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
         @JsonDeserialize(using = LocalDateTimeDeserializer.class)
         LocalDateTime movieShowTime,
         
         String seatNumber,
         
         @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
         @JsonDeserialize(using = LocalDateTimeDeserializer.class)
         LocalDateTime messageGeneratedOn,
         
         String users)
 {}

 package in.learnjavaskills.bookingprocessservice.dto;

 import com.fasterxml.jackson.annotation.JsonFormat;

 import java.time.LocalDateTime;

 public record MovieTicketBookedSuccessfully(String movieId,
         String user,
         String moviePassCode,
         
         @JsonFormat(pattern = "yyyy-MM-dd HH:mm:ss")
         LocalDateTime messageGeneratedOn)
 {}

What exactly is KStream?

KStream<K,V> is an interface in the org.apache.kafka.streams.kstream package.

KStream is an abstraction of a record stream of KeyValue pairs, i.e., each record in the real world is an individual entity or event. For example, if user A purchases two goods, item1 and item2, there may be two records <K:item1> and <K:item2>, in the stream.

What exactly is KeyValue?

The class KeyValue<K, V> is part of the org.apache.kafka.streams package.

For each Kafka Streams record, a key-value pair is defined. If the record is directly from a Kafka topic, its key/value is the message key/value.

Kafka Broker Configuration in the Booking Process Service

First, in the application.properties file, we must configure our Kafka broker node, and I have named my service booking-process-service-application.

 # application name
 spring.application.name = booking-process-service-application

 # broker server node
 spring.cloud.stream.kafka.binder.brokers = 54.146.205.212:9092

Then you may give spring.cloud.function.definition=processMovieTicketBooking definitions. With the binder's functional composition support

 # kafka binder name - method name of your function
 spring.cloud.function.definition = processMovieTicketBooking

Enable the kafka consumer group to consume messages in a group. This property allows you to consume the message in the form of a group name; in our example, booking-process-service-application.

 # kafka consumer group name
 spring.cloud.stream.bindings.processMovieTicketBooking.group = ${spring.application.name}

The following are the Kafka topic name binder and deserialization configuration properties:.

 # kafka topic name
 spring.cloud.stream.bindings.processMovieTicketBooking-in-0.destination = movie-ticket-request-topic
 spring.cloud.stream.bindings.processMovieTicketBooking-out-0.destination = movie-ticket-booked-topic

 # deserialization configuration
 spring.cloud.stream.kafka.bindings.processMovieTicketBooking.in-0.consumer.configuration.key.deserializer = org.apache.kafka.common.serialization.StringDeserializer
 spring.cloud.stream.kafka.bindings.processMovieTicketBooking.in-0.consumer.configuration.value.deserializer = org.springframework.kafka.support.serializer.JsonDeserializer

 spring.cloud.stream.kafka.bindings.processMovieTicketBooking.in-0.producer.configuration.key.deserializer = org.apache.kafka.common.serialization.StringDeserializer
 spring.cloud.stream.kafka.bindings.processMovieTicketBooking.in-0.producer.configuration.value.deserializer = org.springframework.kafka.support.serializer.JsonDeserializer

Dependencies

 dependencies {
	implementation 'org.apache.kafka:kafka-streams'
	implementation 'org.springframework.cloud:spring-cloud-stream'
	implementation 'org.springframework.cloud:spring-cloud-stream-binder-kafka-streams'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'org.springframework.cloud:spring-cloud-stream-test-binder'
	// required this dependencies to deserialize java 8 date/time
	implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.16.0'
 }

Update Booked Status Service

Finally, using the Spring Cloud Stream and the Kafka Stream, we will develop another service to consume messages from the Kafka topic.

In this service, we will utilize the Java 8 consumer interface to consume the message from the Kafak topic in a functional approach, similar to the previous example.

Let's utilize Java 8's Consumer Interface to consume messages from the Kafka topic. Because it receives just one argumentase input and does not give any response, the consumer interface is utilized to consume the message.

Create a bean called consumeMovieTickedBookingMessage that we will associate with in the properties file to consume the message from Kafka.

 package in.learnjavaskills.updatebookedstatusservice.function;

 import in.learnjavaskills.updatebookedstatusservice.dto.MovieTicketBookedSuccessfully;
 import in.learnjavaskills.updatebookedstatusservice.repository.MovieTicketRepository;
 import org.apache.kafka.streams.kstream.KStream;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.context.annotation.Bean;
 import org.springframework.context.annotation.Configuration;

 import java.util.function.Consumer;

 @Configuration
 public class MovieTicketBookingConsumer
 {
    private final MovieTicketRepository movieTicketRepository;

    @Autowired
    public MovieTicketBookingConsumer(MovieTicketRepository movieTicketRepository) {
        this.movieTicketRepository = movieTicketRepository;
    }

    @Bean("consumeMovieTickedBookingMessage")
    public Consumer<KStream<String, MovieTicketBookedSuccessfully>> consumeMovieTickedBookingMessage() {
        return movieTicketBookingMessageKStream -> {
          movieTicketBookingMessageKStream.foreach((messageKey, message) -> {
              System.out.println("Message Key : " + messageKey + " Message : " + message.toString());
              // Saving the successfully booked movie ticked into the data base.
              movieTicketRepository.save(message);
          });
        };
    }
 }

Kafka Broker Configuration in the Update Booked Status Service

 # application name
 spring.application.name = update-booked-status-service-application

 # broker server node
 spring.cloud.stream.kafka.binder.brokers = 54.146.205.212:9092

 # kafka binder name - method name of your function
 spring.cloud.function.definition = consumeMovieTickedBookingMessage

 # kafka consumer group name
 spring.cloud.stream.bindings.consumeMovieTickedBookingMessage.group = ${spring.application.name}

 # kafka topic name
 spring.cloud.stream.bindings.consumeMovieTickedBookingMessage-in-0.destination = movie-ticket-booked-topic


 # deserialization configuration
 spring.cloud.stream.kafka.bindings.consumeMovieTickedBookingMessage.in-0.consumer.configuration.key.deserializer = org.apache.kafka.common.serialization.StringDeserializer
 spring.cloud.stream.kafka.bindings.consumeMovieTickedBookingMessage.in-0.consumer.configuration.value.deserializer = org.springframework.kafka.support.serializer.JsonDeserializer


 # adding the trusted package to deserialize the JSON data else it will throw to add the trusted package exception
 spring.cloud.stream.kafka.streams.binder.configuration.spring.json.trusted.packages = '*'
 spring.cloud.stream.kafka.streams.binder.configuration.spring.json.use.type.headers = false

Dependencies

 dependencies {
	implementation 'org.apache.kafka:kafka-streams'
	implementation 'org.springframework.cloud:spring-cloud-stream'
	implementation 'org.springframework.cloud:spring-cloud-stream-binder-kafka-streams'
	testImplementation 'org.springframework.boot:spring-boot-starter-test'
	testImplementation 'org.springframework.cloud:spring-cloud-stream-test-binder'
	// required this dependencies to deserialize java 8 date/time
	implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.16.0'
 }

The source code is available on (getButton) #text=(GitHub) #icon=(download) #color=(#000000)

Conclussion

Spring CLoud Stream 3.0 encourages us to consume or produce messages in the Kafak topic using the functional approach.We covered how to use the Spring Cloud Stream with the Kafka Stream in this presentation.

As you can see in this example, I utilized the IP address of the Kafak broker server node, however this is not the best practice. You should always utilize host names rather than IP addresses.

Keep learning and keep growing.