1. ev
  2. Soru-cevap
  3. Camel-Jetty http proxy + büyük yanıt verileri dizisi sorunu: Tamponlama kapasitesi

Camel-Jetty http proxy + büyük yanıt verileri dizisi sorunu: Tamponlama kapasitesi

2016-04-11 24 views
3

iskelesi uzmanı aştı, büyük veriyi http proxy gibi deve iskelesi üzerinden geri alırken hata yaşıyorum, herhangi bir deneyim/fikir/öneri memnuniyetle karşılıyor. RouteBuilder kodu:Camel-Jetty http proxy + büyük yanıt verileri dizisi sorunu: Tamponlama kapasitesi

from("servlet:balancer?matchOnUriPrefix=true") 
.to("jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?" + "bridgeEndpoint=true" 
              + "&disableStreamCache=true" 
              + "&throwExceptionOnFailure=false" 
              + "&bufferSize=1024000000" 
              + "&inboundBufferSize=1024000000" 
              + "&outboundBufferSize=1024000000" 
              + "&eagerCheckContentAvailable=true" 
              + "&httpClient.responseBufferSize=1024000000&responseBufferSize=1024000000&chunked=false")

ENV:

<jdk.version>1.8</jdk.version> 
<camel.version>2.17.0</camel.version> 
<spring.version>4.2.4.RELEASE</spring.version> 
<spring.security.version>4.0.4.RELEASE</spring.security.version>
http tabanlı dinlenme hizmet dönüş geri json veri 5000 kayıtları, ben, tepki tampon boyutunu maalesef hala ayarlamak için birçok yol denedi

Got hatası

[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e 
[email protected][p=40262,l=47776,c=1024000000,r=7514]={umn12":"data4782"...4875","\r\n2000\r\n<<<column10":"data48...16":"data4892",>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser       1232] - parseNext s=CHUNK [email protected][p=47776,l=47776,c=1024000000,r=0]={umn12":"data4782"...16":"data4892",<<<>>>data3198","column...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      158] - Parsed false, remaining 0 HttpParser{s=CHUNK,2096474 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint       142] - filled 31280 [email protected]{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,3/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      113] - Read 31280 bytes [email protected][p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} from [email protected]{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,0/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser       1232] - parseNext s=CHUNK [email protected][p=0,l=31280,c=1024000000,r=31280]={<<<"column17":"data4...ata4964","colum>>>54","column16":"d...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e 
[email protected][p=0,l=678,c=1024000000,r=678]={<<<"column17":"data4...a4894","column8>>>\r\n2000\r\n":"data48...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser       1587] - CHUNK --> CHUNKED_CONTENT 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser       1587] - CHUNKED_CONTENT --> CHUNK_SIZE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser       1587] - CHUNK_SIZE --> CHUNK 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      314] - Response content HttpResponse[HTTP/1.1 200 OK]@54e3183e 
[email protected][p=686,l=8878,c=1024000000,r=8192]={"column17":"data4...column8\r\n2000\r\n<<<":"data4894","col..."column6":"data>>>\r\n2000\r\n4913","co...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange      228] - Failed [email protected] req=TERMINATED/[email protected] res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity [email protected]: req=false/rsp=true java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser       1555] - close HttpParser{s=CHUNK,2105344 of -1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.http.HttpParser       1587] - CHUNK --> CLOSED 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      514] - Response failure HttpResponse[HTTP/1.1 200 OK]@54e3183e [email protected] req=TERMINATED/[email protected] res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity [email protected] on [email protected]([email protected] req=TERMINATED/[email protected] res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity [email protected])[[email protected](req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],[email protected](rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      529] - Concurrent failure: response termination skipped, performed by helpers 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange      259] - Aborted (true) while active [email protected] req=TERMINATED/[email protected] res=COMPLETED/java.lang.IllegalArgumentException: Buffering capacity [email protected]: java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      541] - State update failed: TRANSIENT -> CONTENT: FAILURE 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpExchange      210] - Terminated response for [email protected] req=TERMINATED/[email protected] res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity [email protected], result: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      442] - Response complete HttpResponse[HTTP/1.1 200 OK]@54e3183e 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpChannel       89] - [email protected] req=TERMINATED/[email protected] res=TERMINATED/java.lang.IllegalArgumentException: Buffering capacity [email protected] disassociated true from [email protected](exchange=null)[[email protected](req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],[email protected](rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.ConnectionPool      257] - Connection removed [email protected](l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[[email protected](exchange=null)[[email protected](req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],[email protected](rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] - pooled: 0 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint       75] - oshut [email protected]{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,Open,in,out,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP   154] - [email protected](l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[[email protected](exchange=null)[[email protected](req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],[email protected](rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] oshut 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint      108] - onClose [email protected]{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.ChannelEndPoint       115] - close [email protected]{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,in,OSHUT,-,-,4/0,HttpConnectionOverHTTP}{io=0,kio=0,kro=1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.SelectorManager       756] - Destroyed [email protected]{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractConnection      210] - onClose [email protected](l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[[email protected](exchange=null)[[email protected](req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],[email protected](rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.io.AbstractEndPoint      108] - onClose [email protected]{vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999<->51324,CLOSED,ISHUT,OSHUT,-,-,5/0,HttpConnectionOverHTTP}{io=0,kio=-1,kro=-1} 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.http.HttpConnectionOverHTTP   157] - [email protected](l:/10.114.146.192:51324 <-> r:vm-a38e-8f19.nam.nsroot.net/10.42.108.76:19999,closed=true)[[email protected](exchange=null)[[email protected](req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],[email protected](rsp=FAILURE,failure=java.lang.IllegalArgumentException: Buffering capacity exceeded)[HttpParser{s=CLOSED,2105344 of -1}]]] closed 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      450] - Request/Response failed: Result[HttpRequest[GET /balancer/restservice/largedata HTTP/1.1]@66fc8f65 > HttpResponse[HTTP/1.1 200 OK]@54e3183e] java.lang.IllegalArgumentException: Buffering capacity exceeded 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.DefaultErrorHandler    153] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). On delivery attempt: 0 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [DEBUG] [org.apache.camel.processor.Pipeline       78] - Message exchange has failed: so breaking out of pipeline for exchange: Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8] Exception: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
[2016/04/11 18:32:09 CST] [ERROR] [org.apache.camel.processor.DefaultErrorHandler    204] - Failed delivery for (MessageId: ID-APACCNDLBIW2480-51289-1460370708686-0-7 on ExchangeId: ID-APACCNDLBIW2480-51289-1460370708686-0-8). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 

Message History 
--------------------------------------------------------------------------------------------------------------------------------------- 
RouteId    ProcessorId   Processor                  Elapsed (ms) 
[route1   ] [route1   ] [servlet:/balancer?matchOnUriPrefix=true          ] [  3135] 
[route1   ] [policy1   ] [policy[null]                 ] [  3136] 
[route1   ] [to1    ] [jetty:http://vm-a38e-8f19.nam.nsroot.net:19999/balancer/?bridgeEndpoint=true&d] [  3133] 

Stacktrace 
--------------------------------------------------------------------------------------------------------------------------------------- 
org.apache.camel.CamelExchangeException: JettyClient failed cause by: Buffering capacity exceeded. Exchange[ID-APACCNDLBIW2480-51289-1460370708686-0-8]. Caused by: [java.lang.IllegalArgumentException - Buffering capacity exceeded] 
    at org.apache.camel.component.jetty9.JettyContentExchange9.doTaskCompleted(JettyContentExchange9.java:155) 
    at org.apache.camel.component.jetty9.JettyContentExchange9$2.onComplete(JettyContentExchange9.java:221) 
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193) 
    at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185) 
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:453) 
    at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:434) 
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:360) 
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:254) 
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:38) 
    at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1499) 
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1272) 
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:156) 
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:117) 
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69) 
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89) 
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122) 
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) 
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) 
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded 
    at org.eclipse.jetty.client.util.BufferingResponseListener.onContent(BufferingResponseListener.java:114) 
    at org.eclipse.jetty.client.api.Response$Listener$Adapter.onContent(Response.java:245) 
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:124) 
    at org.eclipse.jetty.client.ResponseNotifier.access$100(ResponseNotifier.java:35) 
    at org.eclipse.jetty.client.ResponseNotifier$ContentCallback.process(ResponseNotifier.java:274) 
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246) 
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208) 
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:117) 
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:322) 
    ... 13 more 
[2016/04/11 18:32:09 CST] [DEBUG] [org.eclipse.jetty.client.HttpReceiver      158] - Parsed true, remaining 22402 HttpParser{s=CLOSED,2105344 of -1}

kaynak

2016-04-11 Lori Sun

+0

Jetty'nin hiçbir şeyi arabelleğe almamasını bilmiyorum. Mümkün olduğundan emin değilim. Proxy rotanızda 'http4' bileşenini üretici olarak deneyebilir misiniz? Camel 2.17.0'da, akış önbelleği devre dışı bırakılmışsa yanıt akışının [önbelleklenmeyeceği] (https://issues.apache.org/jira/browse/CAMEL-7638) ek faydası ile birlikte gelir. – Ralf

+0

Başka bir nedenden ötürü, deve-jetty kullanmam gerekiyor, yine de, hiçbir şey önbelleğe almayı nasıl anlayacağımı anlayacağım, ilk tercih bu :), yanıt önbellek arabelleği boyutunu ikincil olarak değiştir ... –

cevap

3

nedeni:: hizmet tarafı dönüş < 1000 kayıtları ne zaman çalışma, çalışıyor deve jetty9-2.17.0.jar bir sınırlama, org.apache.camel.component.j vardır etty9.JettyContentExchange9 # maxLength 2MB (* 1024 2 * 1024) varsayılan vardır BufferingResponseListener yılında 

BufferingResponseListener responseListener = new BufferingResponseListener() { 

    @Override 
    public void onComplete(Result result) { 
     if (result.isFailed()) { 
      doTaskCompleted(result.getFailure()); 
     } else { 
      onResponseComplete(result, getContent(), getMediaType()); 
     } 
    } 
};

gönderin. Tek parametre kurucu: BufferingResponseListener (int maxLength), maxLength'u ayarlamak için sadece bir yoldur. ama deve varsayılan bir kurucu kullanır, bu yüzden 2 MB'den büyük tüm yanıtlar böyle bir hata alır.

kaynak

2016-04-14 08:00:21

İlgili konular

 İlgili konular