atom feed8 messages in org.apache.hc.httpclient-usersRE: HTTP Response contains headers last
FromSent OnAttachments
Morris, ChrisSep 26, 2007 11:53 am 
Roland WeberSep 27, 2007 8:32 am 
Morris, ChrisSep 27, 2007 8:48 am 
Roland WeberSep 27, 2007 9:16 am 
Morris, ChrisSep 27, 2007 9:24 am 
Roland WeberSep 27, 2007 9:55 am 
Morris, ChrisSep 27, 2007 10:03 am 
Roland WeberSep 27, 2007 10:57 pm 
Subject:RE: HTTP Response contains headers last
From:Morris, Chris (Chri@epsilon.com)
Date:Sep 27, 2007 8:48:59 am
List:org.apache.hc.httpclient-users

Thanks Roland.

I tried treating them as footers to no avail. I still don't get access to the actual body of the response.

Here's the wire log. As you can see from below, the "data" portion of the response (which should be considered the body) is being skipped over to get to the headers. The problem is that attempting to grab the data as headers, footers or body is not working. It's like the method is throwing those lines away (but the wire output thinks they are header rows).

[INFO] DreamAPIRequest - -Extracting activity for campaign: test mailing: test01 [DEBUG] HttpClient - -Java version: 1.5.0_10 [DEBUG] HttpClient - -Java vendor: Sun Microsystems Inc. [DEBUG] HttpClient - -Java class path: C:\Documents and Settings\cmorris\workspace\AMI_2.0_Local\target;C:\Documents and Settings\cmorris\workspace\AMI_2.0_Local\target\classes;C:\Documents and Settings\cmorris\.m2\repository\commons-codec\commons-codec\1.2\commons- codec-1.2.jar;C:\Documents and Settings\cmorris\.m2\repository\commons-httpclient\commons-httpclient\3. 0.1\commons-httpclient-3.0.1.jar;C:\Documents and Settings\cmorris\.m2\repository\commons-io\commons-io\1.3.1\commons-io-1 .3.1.jar;C:\Documents and Settings\cmorris\.m2\repository\commons-logging\commons-logging\1.0.3\co mmons-logging-1.0.3.jar;C:\Documents and Settings\cmorris\.m2\repository\commons-vfs\commons-vfs\1.0\commons-vfs- 1.0.jar;C:\Documents and Settings\cmorris\.m2\repository\jdom\jdom\1.0\jdom-1.0.jar;C:\Documents and Settings\cmorris\.m2\repository\junit\junit\3.8.1\junit-3.8.1.jar;C:\Doc uments and Settings\cmorris\workspace\AMI_2.0_Local\jar\commons-httpclient-3.0.1.ja r;C:\Documents and Settings\cmorris\workspace\AMI_2.0_Local\jar\commons-logging-1.0.3.jar;C :\Documents and Settings\cmorris\workspace\AMI_2.0_Local\jar\commons-vfs-20070719.jar;C: \Documents and Settings\cmorris\workspace\AMI_2.0_Local\jar\jdom-1.0.jar;C:\Documents and Settings\cmorris\workspace\AMI_2.0_Local\jar\xerces.jar;C:\Documents and Settings\cmorris\workspace\AMI_2.0_Local\jar\commons-codec-1.2.jar;C:\Do cuments and Settings\cmorris\workspace\AMI_2.0_Local\jar\commons-net-1.4.1.jar;C:\Do cuments and Settings\cmorris\workspace\AMI_2.0_Local\jar\jsch-0.1.32.jar [DEBUG] HttpClient - -Operating system name: Windows XP [DEBUG] HttpClient - -Operating system architecture: x86 [DEBUG] HttpClient - -Operating system version: 5.1 [DEBUG] HttpClient - -SUN 1.5: SUN (DSA key/parameter generation; DSA signing; SHA-1, MD5 digests; SecureRandom; X.509 certificates; JKS keystore; PKIX CertPathValidator; PKIX CertPathBuilder; LDAP, Collection CertStores) [DEBUG] HttpClient - -SunRsaSign 1.5: Sun RSA signature provider [DEBUG] HttpClient - -SunJSSE 1.5: Sun JSSE provider(PKCS12, SunX509 key/trust factories, SSLv3, TLSv1) [DEBUG] HttpClient - -SunJCE 1.5: SunJCE Provider (implements RSA, DES, Triple DES, AES, Blowfish, ARCFOUR, RC2, PBE, Diffie-Hellman, HMAC) [DEBUG] HttpClient - -SunJGSS 1.0: Sun (Kerberos v5) [DEBUG] HttpClient - -SunSASL 1.5: Sun SASL provider(implements client mechanisms for: DIGEST-MD5, GSSAPI, EXTERNAL, PLAIN, CRAM-MD5; server mechanisms for: DIGEST-MD5, GSSAPI, CRAM-MD5) [DEBUG] DefaultHttpParams - -Set parameter http.useragent = Jakarta Commons-HttpClient/3.0.1 [DEBUG] DefaultHttpParams - -Set parameter http.protocol.version = HTTP/1.1 [DEBUG] DefaultHttpParams - -Set parameter http.connection-manager.class = class org.apache.commons.httpclient.SimpleHttpConnectionManager [DEBUG] DefaultHttpParams - -Set parameter http.protocol.cookie-policy = rfc2109 [DEBUG] DefaultHttpParams - -Set parameter http.protocol.element-charset = US-ASCII [DEBUG] DefaultHttpParams - -Set parameter http.protocol.content-charset = ISO-8859-1 [DEBUG] DefaultHttpParams - -Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@1198891 [DEBUG] DefaultHttpParams - -Set parameter http.dateparser.patterns = [EEE, dd MMM yyyy HH:mm:ss zzz, EEEE, dd-MMM-yy HH:mm:ss zzz, EEE MMM d HH:mm:ss yyyy, EEE, dd-MMM-yyyy HH:mm:ss z, EEE, dd-MMM-yyyy HH-mm-ss z, EEE, dd MMM yy HH:mm:ss z, EEE dd-MMM-yyyy HH:mm:ss z, EEE dd MMM yyyy HH:mm:ss z, EEE dd-MMM-yyyy HH-mm-ss z, EEE dd-MMM-yy HH:mm:ss z, EEE dd MMM yy HH:mm:ss z, EEE,dd-MMM-yy HH:mm:ss z, EEE,dd-MMM-yyyy HH:mm:ss z, EEE, dd-MM-yyyy HH:mm:ss z] [INFO] HttpAPIRequest - -postRequest posting to: http://dweb.bfi0.com:9000/api/ActivityDownload.mpl [INFO] HttpAPIRequest - -HTTP Post contains the following parameters: [INFO] HttpAPIRequest - -parameter: headers_first = 0 [INFO] HttpAPIRequest - -parameter: client_name = xxxxxxxxxxxx [INFO] HttpAPIRequest - -parameter: client_token = xxxxxxxxxxxxx [INFO] HttpAPIRequest - -parameter: output_field_delimiter = || [INFO] HttpAPIRequest - -parameter: campaign_name = test [INFO] HttpAPIRequest - -parameter: mailing_name = test01 [INFO] HttpAPIRequest - -parameter: start_date = 20070901 [INFO] HttpAPIRequest - -parameter: end_date = 20070930 [INFO] HttpAPIRequest - -parameter: ret_mod_name = Y [INFO] HttpAPIRequest - -parameter: ret_link_name = Y [INFO] HttpAPIRequest - -parameter: profile_field_names = CUST_ID [DEBUG] HttpConnection - -Open connection to xxxx.xxxx.xxx:9000 [DEBUG] header - ->> "POST /api/ActivityDownload.mpl HTTP/1.1[\r][\n]" [DEBUG] HttpMethodBase - -Adding Host request header [DEBUG] HttpMethodBase - -Default charset used: ISO-8859-1 [DEBUG] HttpMethodBase - -Default charset used: ISO-8859-1 [DEBUG] header - ->> "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7[\r][\n]" [DEBUG] header - ->> "Keep-Alive: 300[\r][\n]" [DEBUG] header - ->> "Connection: keep-alive[\r][\n]" [DEBUG] header - ->> "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai n;q=0.8,image/png,*/*;q=0.5[\r][\n]" [DEBUG] header - ->> "Accept-Language: en-us,en;q=0.5[\r][\n]" [DEBUG] header - ->> "Accept-Encoding: gzip,deflate[\r][\n]" [DEBUG] header - ->> "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7[\r][\n]" [DEBUG] header - ->> "Host: xxxx.xxx.xxx:9000[\r][\n]" [DEBUG] header - ->> "Content-Length: 238[\r][\n]" [DEBUG] header - ->> "Content-Type: application/x-www-form-urlencoded[\r][\n]" [DEBUG] header - ->> "[\r][\n]" [DEBUG] EntityEnclosingMethod - -Request body sent [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||1||20070 911 111040||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||1||20070 911 111602||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 911 111602||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 911 111602||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 911 111603||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||1||2007091 1 110746||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||1||2007091 1 110748||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||1||2007091 1 110749||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||SEND||H||S||1||||||||1||2007091 1 110751||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||1||2007091 1 110752||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||R||280988||http://www. epsiloninteractive.com||TOPLEVEL||test_linkname_html||1||20070911 111120||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 912 152737||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||R||280988||http://www. epsiloninteractive.com||TOPLEVEL||test_linkname_html||0||20070912 152742||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||1||20070 913 221549||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||R||280988||http://www. epsiloninteractive.com||TOPLEVEL||test_linkname_html||1||20070913 221553||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||R||280988||http://www. epsiloninteractive.com||TOPLEVEL||test_linkname_html||0||20070913 221602||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 914 102710||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 918 165151||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||1||20070 918 170651||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 918 170651||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||1||20070 918 172650||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 918 172650||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 918 172743||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||T||C||DEF||||||||1||20070 918 174925||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||HVWB||T||O||DEF||||||||1||20070 918 175044||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||0||2007091 8 170117||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||0||2007091 8 170122||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||0||2007091 8 170124||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||SEND||H||S||1||||||||0||2007091 8 170125||[\r][\n]" [DEBUG] header - -<< "test01||ema@foo.com||ema@foo.com||SEND||H||S||1||||||||0||2007091 8 170126||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 918 190149||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||R||280988||http://www. epsiloninteractive.com||TOPLEVEL||test_linkname_html||0||20070918 190154||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||R||331121||http://www. epsiloninteractive.com||TEST_MODULE||test_module_link_name_html||1||2007 0918 190158||[\r][\n]" [DEBUG] header - -<< "test01||Ema@foo.com||ema@foo.com||HVWB||H||G||DEF||||||||0||20070 919 113605||[\r][\n]" [DEBUG] header - -<< "HTTP/1.1 200 OK[\r][\n]" [DEBUG] header - -<< "Date: Thu, 27 Sep 2007 15:38:28 GMT[\r][\n]" [DEBUG] header - -<< "Server: Apache[\r][\n]" [DEBUG] header - -<< "BFI-Status: OK[\r][\n]" [DEBUG] header - -<< "BFI-Status-String: [\r][\n]" [DEBUG] header - -<< "BFI-Number-Records: 34[\r][\n]" [DEBUG] header - -<< "Connection: close[\r][\n]" [DEBUG] header - -<< "Transfer-Encoding: chunked[\r][\n]" [DEBUG] header - -<< "Content-Type: text/plain[\r][\n]" [WARN] HttpMethodBase - -Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.test||

[DEBUG] HttpMethodBase - -Buffering response body [DEBUG] HttpMethodBase - -Should close connection in response to directive: close [DEBUG] HttpConnection - -Releasing connection back to connection manager. [INFO] HttpAPIRequest - -postRequest received the following response: HTTP/1.1 200 OK [DEBUG] HttpAPIRequest - -response header: Date: Thu, 27 Sep 2007 15:38:28 GMT

[DEBUG] HttpAPIRequest - -response header: Server: Apache

[DEBUG] HttpAPIRequest - -response header: BFI-Status: OK

[DEBUG] HttpAPIRequest - -response header: BFI-Status-String:

[DEBUG] HttpAPIRequest - -response header: BFI-Number-Records: 34

[DEBUG] HttpAPIRequest - -response header: Connection: close

[DEBUG] HttpAPIRequest - -response header: Transfer-Encoding: chunked

[DEBUG] HttpAPIRequest - -response header: Content-Type: text/plain

[DEBUG] HttpAPIRequest - -response body:

[INFO] DreamAPIRequest - -BFI-Status: OK [INFO] DreamAPIRequest - -BFI-Status-String: [INFO] DreamAPIRequest - -BFI-Number-Records: 34 [INFO] FileUtility - -Writing string buffer to File: email_response.datAppend Flag: false

Chris Morris

Sorry about the footer... I have no control over what the exchange server appends to my messages on my behalf...

-----Original Message----- From: Roland Weber [mailto:ossf@dubioso.net] Sent: Thursday, September 27, 2007 9:33 AM To: HttpClient User Discussion Subject: Re: HTTP Response contains headers last

Morris, Chris wrote:

I'm working with an API that returns the HTTP Response Headers after the response body. This confuses the HTTPClient and PostMethod into thinking that there is no body.

We have code to handle footers in ChunkedInputStream. However, this is the first time that I encounter footers in the wild, so maybe that code is buggy. Could you please post a wire log, including headers and content? Feel free to obfuscate any sensitive information and cut the contents. http://jakarta.apache.org/httpcomponents/httpclient-3.x/logging.html

Is there a way for me to get direct access to the response input stream so that I can parse it out myself?

You can hack through the code of course. The socket's stream is not accessible directly, because the ChunkedInputStream has to parse the "chunked" encoding. Trailers are only allowed with chunked encoding. If you have to hack through the code anyway, you should rather fix what is broken than work around what is already there.

The information contained in this communication is confidential, and is intended only for the sole use of the recipient named above. [...]

Always funny to read these footers on mails sent to public mailing lists...

cheers, Roland