java.lang.OutOfMemoryError when I have 12 cores and 48GB of RAM

These days, virtual machines are very popular in customers' sites. Almost 80-90% of my customers require us to install OpenAM/OpenDJ on their VMs.

But sometimes, it can get really annoying if the virtual machines show errors that are not useful in debugging!!

How is it possible that I can run out of memory when I was given a 12 cores + 48 GB RAM VM?



2012-11-14 10:42:00,184 INFO  [javax.enterprise.resource.webservices.rpc.server.http] (http-172.19.176.116-8280-1) JAXRPCSERVLET14: JAX-RPC servlet initializing
2012-11-14 10:42:25,503 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-3,172.19.176.116:8100) failure passing message down
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
        at org.jgroups.protocols.TP.down(TP.java:1175)
        at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2308)
        at org.jgroups.protocols.Discovery.down(Discovery.java:374)
        at org.jgroups.protocols.MERGE2.down(MERGE2.java:175)
        at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:381)
        at org.jgroups.protocols.FD.down(FD.java:315)
        at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:95)
        at org.jgroups.protocols.BARRIER.down(BARRIER.java:107)
        at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:803)
        at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:604)
        at org.jgroups.protocols.UNICAST.down(UNICAST.java:455)
        at org.jgroups.protocols.pbcast.STABLE$1.run(STABLE.java:644)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

Obviously, there is some mis-match in how the virtual machine is setup versa that with the physical hardware.

[webadm@tmsappr5 ~]$ jboss-adm/SSOADMIN_node1 stop
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: Resource temporarily unavailable
JBOSS_CMD_START = cd /data/webserver/jboss-eap-5.1/jboss-as/bin; /data/webserver/jboss-eap-5.1/jboss-as/bin/run.sh -c SSOADMIN_node1 -P /data/webdata/SSOADMIN_node1/server.properties
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable
jboss-adm/SSOADMIN_node1: fork: retry: Resource temporarily unavailable

.