OutOfMemoryError (OOM) Concept in Java AS for SAP NetWeaver 7.1 and higher
Overview:
A common error that causes crash of AS Java is OutOfMemoryError, i.e, OOM error.
Although the error name indicates some problem with Memory, there are actually multiple possibilities and hence solution varies.
In order to find the cause of OutOfMemory error, open std_serverX.out file which covers the time point when OutOfMemoryError occurred.
There will find some details about OutOfMemoryError in below pattern:
“java.lang.OutOfMemoryError: <Detailed Reason>”
We can see below type of OOM error in above format.
1. java.lang.OutOfMemoryError: Java heap space
2. java.lang.OutOfMemoryError: PermGen space
3. java.lang.OutOfMemoryError: Requested array size exceeds VM limit
4. java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space?
5. java.lang.OutOfMemoryError: unable to create new native thread
6. java.lang.OutOfMemoryError: GC overhead limit exceeded
1. java.lang.OutOfMemoryError: Java heap space:
We can find “java.lang.OutOfMemoryError: Java heap space” in std_server<N>.out.
This error indicates that an object failed to be allocated in Java heap.
This may indicate inappropriate VM configuration (compared to actual workload), or a memory leak.
As a solution we can check few things:
i) Check VM configuration:
* First the Java system administrator should make sure the VM memory parameter is carefully set.
* The most common parameters are regarding Heap Size (“Max heap size”, -Xms), New Size (-XX:MaxNewSize, -XX:NewSize), and Perm Size (-XX:MaxPermSize, -XXPermSize), etc.
* As of general VM configuration recommendation, refer to Note# 1603093 (and also Note# 723909).
ii) Check defaultTrace file:
Find the entry on corresponding time point:
“java.lang.OutOfMemoryError: Java heap space (failed to allocate XXXXX bytes)”
If the size of bytes (XXXX) is more than 200 million (200MB), the coming request might not be reasonable.
Check the call stack, to see who’s generating such request.
iii) Check for memory leak:
Heap dump will tell us what was occupying the VM memory.
iv) Check ../work/std_server<#>.out file and locate the OutOfMemoryError event and determine which part of the heap size is filled. You will see something like this:
In this example the old generation of the heap memory is filled (2793471K while the total size is 2793472K). So the conclusion is that the total heap size is far from being full, still an OutOfMemoryError occured. It is possible that other areas of the heap gets filled as well.
A possible solution is to increase the heap size (-Xms and -Xmx parameters) to prepare the system for higher load. We have to note here, increasing the heap size is not always a solution. If the root cause is a bug that causes memory leak, all the memory will be filled over time, so obviously increasing the heap will not help in these cases.
2. java.lang.OutOfMemoryError: PermGen space
We can found “java.lang.OutOfMemoryError: PermGen space” in std_server0.out.
This error indicates that perm generation is full, which is the space for class, method and string objects.
As a solution we can check few things:
i) Check VM configurations.
*Make sure you have followed Note# 1603093 (and also Note# 723909) for perms size related recommendation.
*If you are running custom application, consult the developer of this application for estimated perm size requirement and set perm size accordingly.
ii) Collect Class statistics on OutOfMemory error.
*Make sure you have JVM parameter set on server node:
“-XX:+DumpDetailedClassStatisticOnOutOfMemory”
* Class Statistics will be written to std_serverX.out file, when OOM occurs.
“Class loader statistic” and “Class meta data statistic” section could help with troubleshooting.
iii) Collect Heap Dump
* Make sure you have JVM parameter set on server node (Note 1004255):
-XX:+HeapDumpOnOutOfMemoryError
-XX:+HeapDumpOnCtrlBreak
* In case the heap dump cannot be generated automatically, collect heap dump manually instead.
* Heap dump file will be written to below directory by default: /usr/sap/<SID>/<instance>/j2ee/cluster/server<N>/
iv) Collect perm related info with JVMMON (provided by SAP JVM).
* Launch JVMMON by command ‘jvmmon-gui’, which is under ../sapjvm_4/tools/bin/
* Choose the process corresponding to the server process.
* In the ‘Dump’ menu, click on ‘Dump class statistic’, ‘Dump detailed class statistic’, and ‘Dump symbol info’.
* Collect std_serverX.out file. Or you can check from statistics the largest class loaded.
This is sometimes not possible, as this has to be collected before OOM error occurs.
You have to monitor the system closely and collect it when Perm usage is much higher than usual.
3. java.lang.OutOfMemoryError: Requested array size exceeds VM limit:
We can find “java.lang.OutOfMemoryError: Requested array size exceeds VM limit” in std_server0.out.
This error indicates that an object failed to be allocated in Java heap as its size is larger than heap size.
This may indicate incorrect VM configuration (heap size too small), or inappropriate memory request (too huge request).
As a solution we can check few things:
i) Check VM configuration.
Firstly the Java system administrator should make sure the VM memory parameter is carefully set.
Special notice should be paid to applications running on AS Java.
ii) Check the call stack where OutOfMemoryError is thrown, and see who is requesting such large array.
The size array can be abnormally large.
For example, below call stack shows that, BW BEX Web report functionality should be responsible for the single request that demanded ~4GB memory.
4. java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space
we can find “java.lang.OutOfMemoryError: request <size> bytes for <reason>. Out of swap space?” in std_server<N>.out.
This error indicates that VM failed to allocate memory from native heap (OS level).
As a solution we can check few things:
i) Check VM configuration
Specifically, make sure initial heap size is set to same value as max heap size, and initial perm size is set to same value as max perm size.
Therefore AS Java won’t request too large memory after it’s started.
ii) Check physical memory configuration.
* Add physical memory, or SWAP space.
* If physical memory cannot be allocated anyway, decrease -Xms, -Xmx, -PermSize, -MaxPermSize so that VM will allocate less physical memory from OS.
* You have to make sure the new VM memory size is still capable of its job.
* If the server is UNIX based OS, set the user limit to ‘unlimited’. It is suggested to reboot the OS so that this setting can be fully effective.
5. java.lang.OutOfMemoryError: unable to create new native thread
we can find “java.lang.OutOfMemoryError: unable to create new native thread” in std_server<N>.out.
This error indicates that VM failed to allocated more thread from OS.
This usually indicates problem with thread resource, instead of memory resource.
As a solution we can check few things:
i) Check thread configurations.
* Open configtool.
* Check dispatcher settings (on Global level and instance level):
* managers -> ThreadManager -> InitialThreadCount, MaxThreadCount and MinThreadCount.
Check server settings (on Global level and instance level):
* managers -> ApplicationThreadManager -> InitialThreadCount, MaxThreadCount and MinThreadCount
* managers -> ThreadManager -> InitialThreadCount, MaxThreadCount and MinThreadCount.
* Make sure you have set reasonable value for this (default value, or the value suggested by SAP note / you have verified).
ii) Check VM configuration:
Make sure -Xss is the default value (2MB for 64bit OS)
This parameter defines the stack size for each thread.
iii) Check and make sure there’s enough physical memory on OS level
iv) If the server is UNIX based OS, set the user limit to ‘unlimited’. It is suggested to reboot the OS so that this setting can be fully effective.
6. java.lang.OutOfMemoryError: GC overhead limit exceeded
We can find “java.lang.OutOfMemoryError: GC overhead limit exceeded” in std_server0.out.
This error indicates that, the average time spent with Garbage Collection is more than 90% for 5 consecutive Garbage Collection runs, yet none of them freed up more than 10 percent of the Java heap.
As a solution we can check few things:
i) Check VM configuration:
Make sure you have followed Note# 1603093 (and also Note# 723909) for general recommendation.
And if you are running other applications on AS Java, also check, then check their respective memory sizinf note.
ii) Check for memory leak.
Follow Note# 1004255 to set -XX:+HeapDumpOnOutOfMemoryError A restart of Java instance is required.
Heap dump file will be written to below directory by default: /usr/sap/<SID>/<instance>/j2ee/cluster/server<N>/
Analyze the heap dump with Memory Analyzer Tool: 1883568 – How to self analyze a Heap Dump using MAT
NOTE: If above all doesn’t work then open a ticket with SAP and provide below logs file/folder
* Attach work folder
* Attach latest defaultTrace
* Describe memory configuration on OS level
* Attach heap dump analysis report