JVM, tüm CPU'ları, çoğu iş parçacığını BLOCKED olarak kullanır. JVM hatası?

Dün gece, bir sunucu (JBoss 5.1GA, Java (TM) SE Çalışma Zamanı Ortamı (build 1.6.0_20-b02), Java HotSpot (TM) VM-VM üzerinde bir Linux VM'de çalışan 64-Bit Sunucu VM) aniden başladı % 100 CPU kullanmak. Uygulamaya Dikiş, oldukça özel bir J2EE iş uygulaması, bu konuda özel bir şey yok. Bu zamanda yük çok düşüktü.JVM, tüm CPU'ları, çoğu iş parçacığını BLOCKED olarak kullanır. JVM hatası?

İşlem öldürülmeden ve yeniden başlatılmadan önce bir iş parçacığı dökümü almayı başardım (uygulama mevcut olmalıydı). JStack'ın kilitlenme tespitinde bir istisna vardı; Bunun dışında çoğu iş parçacığı BLOCKED idi.

Bu bir JVM hatası mı? Daha önce görmedim. Yeniden başlattıktan sonra her şey tamamlandı, uygulama o sunucuda kullanıldığından (periyodik güncellemeler ile) aylar gibi.

Önerileriniz için teşekkür ederiz.

Biraz düzenlenmiş bir iş parçacığı (SO'nun posta boyutu sınırına sığması için) aşağıda yer almaktadır. Birkaç özdeş ipliği (JBoss/Seam'in içindeki çeşitli iplik havuzlarından) çıkardım ve yığın izlerinden sadece üst satırları bıraktım.

DÜZENLEME: Tam iplik dökümühttp://pastie.org/1083984

[~]$ jstack -F 13553 
Attaching to process ID 13553, please wait... 
Debugger attached successfully. 
Server compiler detected. 
JVM version is 11.3-b02 
Deadlock Detection: 

java.lang.NullPointerException 
     at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426) 
     at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137) 
     at sun.jvm.hotspot.oops.Oop.isA(Oop.java:92) 
     at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93) 
     at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) 
     at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) 
     at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) 
     at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
     at java.lang.reflect.Method.invoke(Method.java:597) 
     at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) 
     at sun.tools.jstack.JStack.main(JStack.java:84) 
Can't print deadlocks:null 
Thread 26208: (state = BLOCKED) 


Thread 25250: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 25249: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 21240: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 21140: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 21139: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 21138: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 19380: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 19377: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 19361: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 

Thread 19343: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 19317: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 18995: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 

Thread 18986: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 

Thread 17659: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 17658: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 17653: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 17631: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14977: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14796: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14609: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14437: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14248: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 14232: (state = BLOCKED) 
- java.util.Arrays.copyOf(char[], int) @bci=1, line=2882 (Compiled frame) 
- java.lang.AbstractStringBuilder.expandCapacity(int) @bci=33, line=100 (Compiled frame) 

Thread 14040: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=384 (Interpreted frame) 

Thread 13858: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame) 

Thread 13815: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 

Thread 13814: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13813: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8, line=416 (Interpreted frame) 


Thread 13812: (state = BLOCKED) 
Error occurred during stack walking: 
sun.jvm.hotspot.utilities.AssertionFailure: range check 
     at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32) 
     at sun.jvm.hotspot.runtime.RegisterMap.<init>(RegisterMap.java:109) 
     at sun.jvm.hotspot.runtime.x86.X86RegisterMap.<init>(X86RegisterMap.java:39) 
     at sun.jvm.hotspot.runtime.x86.X86RegisterMap.clone(X86RegisterMap.java:43) 
     at sun.jvm.hotspot.runtime.VFrame.<init>(VFrame.java:37) 
     at sun.jvm.hotspot.runtime.JavaVFrame.<init>(JavaVFrame.java:45) 
     at sun.jvm.hotspot.runtime.CompiledVFrame.<init>(CompiledVFrame.java:43) 
     at sun.jvm.hotspot.runtime.VFrame.newVFrame(VFrame.java:77) 
     at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:129) 
     at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146) 
     at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:231) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76) 
     at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) 
     at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) 
     at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) 
     at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
     at java.lang.reflect.Method.invoke(Method.java:597) 
     at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) 
     at sun.tools.jstack.JStack.main(JStack.java:84) 


Thread 13811: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13810: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13809: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.jms.client.container.ClientConsumer.getMessage(long) @bci=71, line=866 (Interpreted frame) 

Thread 13808: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.jms.client.container.ClientConsumer.getMessage(long) @bci=71, line=866 (Interpreted frame) 

Thread 13807: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.bootstrap.AbstractServerImpl$LifeThread.run() @bci=11, line=866 (Interpreted frame) 

Thread 13806: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 13805: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 13804: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) 

Thread 13803: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) 

Thread 13802: (state = BLOCKED) 
- java.util.Arrays.copyOfRange(char[], int, int) @bci=40, line=3209 (Compiled frame) 
- java.lang.String.<init>(char[], int, int) @bci=65, line=216 (Compiled frame) 
- java.lang.StringBuffer.toString() @bci=13, line=585 (Compiled frame) 
- org.apache.log4j.PatternLayout.format(org.apache.log4j.spi.LoggingEvent) @bci=68, line=506 (Compiled frame) 

Thread 13784: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame) 
- org.jboss.el.util.ReferenceCache$ReferenceQueueRunner.run() @bci=1, line=159 (Interpreted frame) 

Thread 13753: (state = BLOCKED) 
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) 
- org.jboss.ejb3.cache.simple.SimpleStatefulCache$SessionTimeoutTask.block() @bci=11, line=210 (Interpreted frame) 

Thread 13752: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run() @bci=31, line=167 (Interpreted frame) 

Thread 13749: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) 
- org.jboss.remoting.transport.socket.ServerThread.run() @bci=209, line=284 (Interpreted frame) 

Thread 13734: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) 

Thread 13701: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame) 

Thread 13700: (state = BLOCKED) 
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) 
- org.jboss.console.plugins.AOPLister$RefreshPoller.run() @bci=41, line=898 (Interpreted frame) 

Thread 13699: (state = BLOCKED) 
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) 
- org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run() @bci=22, line=1590 (Interpreted frame) 
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) 


Thread 13698: (state = IN_NATIVE) 
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) 

Thread 13678: (state = BLOCKED) 
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) 
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=198 (Interpreted frame) 

Thread 13554: (state = BLOCKED) 

Thread 13560: (state = BLOCKED) 

Thread 13559: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=116 (Compiled frame) 
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=132 (Compiled frame) 
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame) 


Thread 13558: (state = BLOCKED) 
- java.lang.Object.wait(long) @bci=0 (Interpreted frame) 
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame) 
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)

kaynak

2010-08-10 wishihadabettername

Benzersiz engellenmiş parçacıklar için tam yığın izlemeyi görmek istiyorum. Belki tüm boşluğu SO sınırlaması olmadan bir yere götürebilirsin? – Gary

Ben pastie üzerine koydu ve bağlantı ile sorunu güncelledi (http://pastie.org/1083984) – wishihadabettername

Bir olasılık olarak size yığın boşluk bitti olmasıdır. Yığın alanınız tükenirse, JVM, çılgın çöp toplama işlemini gerçekleştirerek büyük bir hızda CPU'yu yakmaya başlayacaktır.

kaynak

2010-08-10 18:03:12

Aynı problem vardı ve aslında çok küçük yığın alanı oldu. Eğer jmap kullanırsanız, yığın alanının aslında sizin probleminiz olup olmadığını anlayabilirsiniz. Komut satırından 'jmap -help ' komutunu çalıştırırsanız, çıkış size kullanılan yığın alanının yüzdesini gösterir. – Roy

Başka bir olasılık bir disk ya da RAM hatasıdır. Özellikle veri transferinde hata düzeltmesi içeren ECC RAM, SAS veya SCSI kullanıyorsanız, bunlar çok nadirdir. Ama olabilirdi!

IBM ve Sun çekirdeği mühendislerinin büyük işletmelere yalnızca bu gibi şeylerle ilgili olarak suçlayabilecekleri destek verdiği bazı garip hataları okudum.

Bu, gerçek nedenini asla keşfedemeyeceğinizi ve yıllarca bir daha asla gerçekleşmeyeceğinizi bilmenizi sağlamanın dışında, size yardımcı olmaz. :)

kaynak

2010-08-10 18:12:19

JVM, tüm CPU'ları, çoğu iş parçacığını BLOCKED olarak kullanır. JVM hatası?

cevap

İlgili konular