0

I am trying to get all the raw request headers from Asynchronous Pluggable Protocol I've implemented. But I can only get a few basic headers using IHttpNegotiate. Such as Accept-Language, Referer. With a tool called HTTP Analyzer these things can be viewed in more detail.

function RetrieveRequestHeaders(const szUrl: PWideChar; const OIProtSink: IInternetProtocolSink): String;
var
  pHttpNeg: IHttpNegotiate;
  Headers: PWideChar;
  HR: HResult;
begin
  Result := '';
  HR := IUnknown_QueryService(OIProtSink, IID_IHttpNegotiate, IID_IHttpNegotiate, pHttpNeg);
  if Succeeded(HR) then
  begin
    Headers := nil;
    HR := pHttpNeg.BeginningTransaction(szUrl, nil, 0, Headers);
    if Succeeded(HR) then
    begin
      Result := Headers;
      CoTaskMemFree(Headers);
    end;
  end;
end;
noseratio
  • 59,932
  • 34
  • 208
  • 486
user3060326
  • 187
  • 2
  • 16
  • I'm not sure I understand the problem. You have implemented a protocol handler - presumably that handler is making the HTTP request. Are you asking which headers you yourself are sending? Don't you know that already? The sink will just tell you which headers are important to it, headers that you won't be able to guess at otherwise (e.g. Accept-Language is a user setting; Referer requires a higher-level context of why this particular request is being made). – Igor Tandetnik Apr 17 '14 at 16:46
  • @IgorTandetnik No I implemented a protocol handler. But I will be doing requests with cURL. But I need to get raw headers so I can launch the request in another thread and then set the response later. I can get raw request headers using `HTTP_QUERY_FLAG_REQUEST_HEADERS` but thats only possible in `BeginngingTransaction`. – user3060326 Apr 17 '14 at 16:53
  • @IgorTandetnik The problem is `BeginningTransaction` is never called because I return `E_PENDING` in `Start(Ex)` – user3060326 Apr 17 '14 at 17:00
  • **Your handler** is supposed to call `BeginningTransaction`. If you don't, then of course it's never called. Who exactly do you expect to call it for you? `HTTP_QUERY_FLAG_REQUEST_HEADERS` is a WinInet flag - but you say you are using curl, not WinInet. I'm confused. You should probably explain your design in more detail. Also state the goal you are trying to achieve - why exactly are you doing all this? – Igor Tandetnik Apr 17 '14 at 17:16
  • @IgorTandetnik [This](http://oi60.tinypic.com/33db47o.jpg) is what I mean. I just want to use `cURL` for requests instead of `WinInet`. Is it possible, if so how? – user3060326 Apr 17 '14 at 17:31
  • 1
    Browser calls `Start[Ex]` on you. You turn around and call `IInternetBindInfo::GetBindInfo` and `BeginningTransaction` on the sink. This gives you enough information to formulate an HTTP request (the sink gives you headers it wants; add any other headers to taste, or just allow libcurl to set up its defaults). Use libcurl API calls to do so. Report progress and data by calling `ReportProgress` and `ReportData` on the sink. – Igor Tandetnik Apr 17 '14 at 17:44
  • @IgorTandetnik Where do I set the response headers? – user3060326 Apr 17 '14 at 17:46
  • The server sets the response headers. If you are asking about request headers, see `curl_easy_setopt(CURLOPT_HTTPHEADER)`. If you are asking how to retrieve response headers, install `CURLOPT_HEADERFUNCTION` callback. – Igor Tandetnik Apr 17 '14 at 18:06
  • @IgorTandetnik I understand but I need to set the response headers back from libcurl to the Webbrowser. `Content-Length, Content-Type` etc. – user3060326 Apr 17 '14 at 18:11
  • 1
    Once you receive the headers, you call `IHttpNegotiate::OnResponse`. You could also implement `IWinInetHttpInfo`, though that would be somewhat difficult - the interface was clearly designed with WinInet in mind. – Igor Tandetnik Apr 17 '14 at 19:16
  • @IgorTandetnik Do I need to use `IWinInetHttpInfo::QueryInfo` to get the AJAX headers or is `IHttpNegotiate::BeginningTransaction` enough? – user3060326 Apr 17 '14 at 23:01
  • What are these "AJAX headers" you speak of, and how do they differ from plain old HTTP headers? AJAX applications send HTTP requests like any others; the protocol handler shouldn't know nor care why the client chose to send this particular request at this particular time. – Igor Tandetnik Apr 17 '14 at 23:17
  • @IgorTandetnik I mean requests done with `xmlhttprequest` in `Javascript` – user3060326 Apr 18 '14 at 00:14
  • They are in no way special as far as the protocol handler is concerned. – Igor Tandetnik Apr 18 '14 at 00:33
  • @IgorTandetnik Can only one thread run in the handler or is it possible to do it via multiple threads? Like `Google-Gears` do it? – user3060326 Apr 24 '14 at 12:32
  • I do not understand the question. What do you mean by "run in the handler"? What does "it" stand for in "do it"? – Igor Tandetnik Apr 24 '14 at 12:45
  • @IgorTandetnik I mean [this](http://stackoverflow.com/questions/2569878/hooking-the-http-https-protocol-in-ie-causes-get-requests-to-be-sequential) – user3060326 Apr 24 '14 at 13:01
  • Well, there's an answer there - use vtable patching to hook up your handler. I've heard about this technique, but never done it myself. – Igor Tandetnik Apr 24 '14 at 18:02
  • @IgorTandetnik There's no code to show how its done.. I doubt its even possible.. – user3060326 Apr 24 '14 at 18:03
  • 1
    It's possible. I've seen it done. [Google Gears](https://code.google.com/p/gears/) (now defunct), in particular, [used this.](https://code.google.com/p/gears/source/browse/trunk/gears/base/ie/vtable_patch.h) Again, I'm not really familiar with the details. – Igor Tandetnik Apr 24 '14 at 18:22
  • @IgorTandetnik Ah it can be done with `Detours` from Microsoft.. – user3060326 Apr 24 '14 at 18:33
  • @IgorTandetnik How to get the `Accept` header from `PassthroughAPP` http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1 – user3060326 May 04 '14 at 10:55
  • The client would report it (and any other request headers it wants) via `pszAdditionalHeaders` parameter of `IHttpNegotiate::BeginningTransaction`. If the client doesn't report this header, then it doesn't particularly care, and you can send `Accept: */*` or omit the header entirely. – Igor Tandetnik May 04 '14 at 13:52
  • @IgorTandetnik I get `*/*` from `BINDSTRING_ACCEPT_MIMES` yet IE reports `text/css, */*` for the same URL. – user3060326 May 04 '14 at 15:48
  • What do you mean by "reports"? Reports where and how? – Igor Tandetnik May 04 '14 at 17:05
  • @IgorTandetnik for the URL `http://www.google.com` APP reports `*/*`. But HTTP Analyzer says `Accept: text/html, application/xhtml+xml, */*` from a real IE browser. – user3060326 May 04 '14 at 17:06
  • Then I'm out of ideas. I do see `BINDSTRING_ACCEPT_MIMES` return something other than `*/*` sometimes, but the values it returns are different from what IE sends (as observed with WireShark). `BeginningTransaction` doesn't appear to ever report `Accept:` header (usually, it's just `Accept-Language:` and `Referer:`). Could be another symptom of "different behavior with custom handler than with built-in handler" issue mentioned earlier. – Igor Tandetnik May 04 '14 at 17:56
  • @IgorTandetnik So its not possible to get these headers? :) – user3060326 May 04 '14 at 18:22
  • It could be possible with vtable patching technique (which, as I already mentioned, I haven't tried myself). I wonder though - why are these small variations in the headers so important to you? It's unlikely that they matter to the actual HTTP servers. – Igor Tandetnik May 04 '14 at 18:31
  • @IgorTandetnik will it affect rendering? – user3060326 May 04 '14 at 21:41
  • @IgorTandetnik Because APP isn't reporting all the headers.. `Stackoverflow` is rendered completely without CSS or Images. Yet google works good. – user3060326 May 05 '14 at 13:43
  • Seems to work for me. I have a test application hosting WebBrowser control, and a passthrough APP (a custom handler that delegates all the work to the built-in handler). StackOverflow renders correctly in this application - images, styles and everything. WireShark shows that all requests are sent with `Accept: */*` – Igor Tandetnik May 05 '14 at 14:20
  • @IgorTandetnik Can you share? :) Are you using `libcurl`? – user3060326 May 05 '14 at 14:21
  • The library based on my (very old by now) Passthrough APP work is available [here](https://github.com/salsita/passthruapp). It doesn't seem to include the test app though. No, it doesn't use libcurl. Like I said, it's a custom handler that delegates most of the work to the built-in handler. – Igor Tandetnik May 05 '14 at 14:35
  • @IgorTandetnik Ofcourse everything works with `INET_E_USE_DEFAULT_PROTOCOLHANDLER` but this isn't a custom handler. This is built in one. Not the thing I need. – user3060326 May 05 '14 at 15:13
  • @IgorTandetnik I think a VTable patch is needed. – user3060326 May 05 '14 at 15:41
  • No, mine doesn't use `INET_E_USE_DEFAULT_PROTOCOLHANDLER`. It's a real custom handler, it does implement all the necessary interfaces and methods - it just happens to implement them by immediately turning around and calling the same method on an instance of the built-in handler it creates (hence "passthrough"). The browser never talks to the built-in handler directly - all calls flow through the custom handler. As I said, my test program sends requests with a different `Accept:` header than those coming from plain IE (as confirmed with a network sniffer). That doesn't seem to affect rendering. – Igor Tandetnik May 05 '14 at 18:32
  • @IgorTandetnik So with your implementation could cURL be used? – user3060326 May 05 '14 at 19:06
  • Not really. It was designed to be pass-through, to sit in the middle between the browser and the built-in handler, to observe (and possibly modify) the detailed information about all HTTP requests. Most of the complexity is there to achieve that. If you don't need pass-through, then you don't need most of that code, so I personally wouldn't choose it as a starting point. Besides, it doesn't solve any of the problems you seem to have anyway (like the issue with `Accept:` header, or the fact that requests apparently don't run in parallel). – Igor Tandetnik May 05 '14 at 21:07
  • @IgorTandetnik Are you sure its not affecting rendering? On left is [my handler](http://oi58.tinypic.com/28wia6r.jpg) on the right is real IE 11. – user3060326 May 06 '14 at 10:59
  • Well, [here's](http://tinypic.com/r/25flmqo/8) what it looks like in my test application. You can even see request and response headers for the first request (witness `Accept: */*`). You must be doing something else wrong. – Igor Tandetnik May 06 '14 at 12:53
  • @IgorTandetnik Well thats impossible. I get the url, request headers from `BeginningTransaction` and then launch request with cURL and set response headers to `OnResponse`. You seem to use `Wininet`. Try with `cURL` and you will not get the same result guaranteed. – user3060326 May 06 '14 at 13:15
  • Well, all I can tell you is what I know. I'm sorry you can't get your project to work, but that's really not my problem. I'm certainly not going to implement your project for you, not even on a dare. I don't have the time nor, quite frankly, the desire. – Igor Tandetnik May 06 '14 at 16:02
  • @IgorTandetnik Well you are compareing `Wininet` and `cURL`. Ofcourse it will work with `Wininet`! You are just monitoring the traffic not making any kind of requests. Not even setting the stream! – user3060326 May 06 '14 at 16:39

1 Answers1

1

IHTTPNegotiate.BeginningTransaction will give you the (additional) headers the browser wants to add to the outgoing request. As the protocol handler you're responsible to create the full outgoing HTTP request header, like you can see with a HTTP analyzer tool.

Stijn Sanders
  • 35,982
  • 11
  • 45
  • 67