Java EE Support Patterns

Java Spring Boot Troubleshooting use cases with FusionReactor

noreply@blogger.com (P-H) — Wed, 06 May 2020 22:23:00 +0000

The following video demonstrates a few troubleshooting use cases with Java and Spring Boot microservices deployed to the Red Hat OpenShift Online Cloud platform.

Cloud Native Platform: RedHat OpenShift Online *v3.11.154
Java version: OpenJDK 11.0.7 *openjdk-11-rhel8/images/1.2-3.1587486933
Application Development Platform: Spring Boot 2.3.0_RC1 *2020/5/1
IDE: Eclipse IDE *2019-09 R (4.13.0)
Monitoring Tool: FusionReactor Cloud offering

Reference

https://app.fusionreactor.io/

Spring Boot High CPU Troubleshooting video with Dynatrace

noreply@blogger.com (P-H) — Mon, 05 Aug 2019 15:34:00 +0000

The following video will provide a tutorial on how to troubleshoot high CPU problems from Spring Boot services deployed to the Red Hat OpenShift Online Cloud platform.

We will first demonstrate how to troubleshoot CPU problems using traditional techniques on Linux OS & containers using a mix of "top" & "jstack" commands.

The second part of the video will demonstrate how to use the Dynatrace monitoring technology to quickly identify which process is consuming high CPU, including code-level view of the top CPU contributors.

Oracle WebLogic Native IO & Java Muxers

noreply@blogger.com (P-H) — Wed, 11 Apr 2018 13:22:00 +0000

This article will provide the complete root cause analysis details and resolution of a Java performance problem affecting a legacy Oracle WebLogic 11g production environment and involving Socket Muxers.

This performance problem was identified while performing a workload migration and performance assessment of a WLS11g environment to RedHat OpenShift container and PaaS platform.

While “Muxers” is an old concept, this post will demonstrate the importance of adequate knowledge of native IO configuration and runtime behavior within a Java EE container such as Oracle WebLogic.

Environment Specifications

Workload location: On-Premises Data Center
Business domain: Telecommunications
NLB & Web server: F5 NLB & Apache
Java EE container: Oracle WebLogic 11g
JDK/JRE: Oracle HotSpot JVM 1.7 64-bit
OS: Solaris 11

APM & Troubleshooting Tools

Cisco AppDynamics
WebLogic 11g Admin console & logs
JVM Thread Dump analysis

References

Oracle® Fusion Middleware Tuning Performance of Oracle WebLogic Server 12c (12.2.1)

Problem & Observations

The problem was first communicated by our production Ops team following recent performance degradation complaints by the end-users under peak load. An initial root cause analysis exercise did reveal the following facts and observations:

Response time spikes were observed on regular basis and especially under peak load.
An analysis of AppDynamics data did expose unexpected delay for inbound traffic via HTTPS.
Processing time of the application web requests (after body/payload received) was found to be optimal and < 1 sec.
An initial review of the WebLogic Threads and JVM Thread Dump did not expose any bottleneck or contention within the application code.
Network packet analysis did not expose any network latency but isolated the response time delay within the WebLogic server tier.

JVM Thread Dump analysis – second pass

Another analysis iteration was performed of the JVM Thread Dump data captured which did reveal the following findings:

As we can see from the above image, it was identified that “Java Muxers” threads were being used for the overall WebLogic Network I/O. In general, it is not recommended enabling the Java Muxers since they offer poor scalability and suboptimal performance vs. native Muxers or more recent NIO Muxers. Java Muxers block on “reads” until there is data to be read from a socket and does not scale well when dealing with a large influx of inbound web requests.

The following Thread stacktrace can be found from the thread dump when using NIO (Oracle WebLogic 12.2.x).

Following the above finding, a review of the WebLogic 11g configuration was performed but did not reveal any problem (native IO enabled). The next phase of the RCA was now to determine why Java Muxers were enabled by WebLogic on start-up.

Root Cause and Solution

The root cause was finally identified following a review of the WebLogic start-up logs.

<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1248787500274> <BEA-000447> <Native IO Disabled. Using Java IO.>

As per above, it was found that Native IO was disabled on start-up due to a problem with the “Performance Pack”, which includes the Native Muxers, falling back on Java IO but still allowing the WebLogic server to start properly.

Furthermore, it was identified that the JVM 1.7 start-up parameters did not include the “-d64” which was confusing & preventing WebLogic from loading the proper 64-bit Performance Pack library, thus disabling Native IO and falling back on the Java Muxers.

Following the implementation of the solution (restoration of the Native Muxers) to the production environment, we could observe a significant improvement of the application performance and improved scalability.

Oracle Open World and Java One 2016 summary later this week

noreply@blogger.com (P-H) — Mon, 17 Oct 2016 10:55:00 +0000

This post is to inform you that I will publish later this week a summary of the highlights last September of my on-site visit in San Francisco and areas of focus for 2017.

I will also publish a few YouTube videos later this month and demonstrate certain trending technologies and some latest Java & JVM troubleshooting techniques.

Please stay tuned.

Thank you.
P-H

Java 8 Performance Optimization - DZone Refcard Update

noreply@blogger.com (P-H) — Mon, 08 Aug 2016 10:16:00 +0000

I am happy to inform you that I published recently an update to the existing refcard on Java Performance Optimization which is now available from DZone. The updated material now better reflects the Java 8 features and provides a dedicated section and guidelines about the JVM Metaspace.

I recommend that you download your FREE copy today.

http://refcardz.dzone.com/refcardz/java-performance-optimization

For now, find below a small snippet:

“By default, the Metaspace memory space is unbounded and will use the available process and/or OS native memory available for dynamic expansions. The memory space is divided into chunks and allocated by the JVM via mmap.We recommend keeping the default, dynamic resize mode as a starting point for simpler sizing combined with close monitoring of your application metadata footprint over time for optimal capacity planning…”

Thank you.

P-H