I am working on a WebCrawler implementation but am facing a strange memory leak in ASP.NET Web API's HttpClient.
So the cut down version is here:
[UPDATE 2]
I found the problem and it is not HttpClient that is leaking. See my answer.
[UPDATE 1]
I have added dispose with no effect:
static void Main(string[] args)
{
int waiting = 0;
const int MaxWaiting = 100;
var httpClient = new HttpClient();
foreach (var link in File.ReadAllLines("links.txt"))
{
while (waiting>=MaxWaiting)
{
Thread.Sleep(1000);
Console.WriteLine("Waiting ...");
}
httpClient.GetAsync(link)
.ContinueWith(t =>
{
try
{
var httpResponseMessage = t.Result;
if (httpResponseMessage.IsSuccessStatusCode)
httpResponseMessage.Content.LoadIntoBufferAsync()
.ContinueWith(t2=>
{
if(t2.IsFaulted)
{
httpResponseMessage.Dispose();
Console.ForegroundColor = ConsoleColor.Magenta;
Console.WriteLine(t2.Exception);
}
else
{
httpResponseMessage.Content.
ReadAsStringAsync()
.ContinueWith(t3 =>
{
Interlocked.Decrement(ref waiting);
try
{
Console.ForegroundColor = ConsoleColor.White;
Console.WriteLine(httpResponseMessage.RequestMessage.RequestUri);
string s =
t3.Result;
}
catch (Exception ex3)
{
Console.ForegroundColor = ConsoleColor.Yellow;
Console.WriteLine(ex3);
}
httpResponseMessage.Dispose();
});
}
}
);
}
catch(Exception e)
{
Interlocked.Decrement(ref waiting);
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine(e);
}
}
);
Interlocked.Increment(ref waiting);
}
Console.Read();
}
The file containing links is available here.
This results in constant rising of the memory. Memory analysis shows many bytes held possibly by the AsyncCallback. I have done many memory leak analysis before but this one seems to be at the HttpClient level.
I am using C# 4.0 so no async/await here so only TPL 4.0 is used.
The code above works but is not optimised and sometimes throws tantrum yet is enough to reproduce the effect. Point is I cannot find any point that could cause memory to be leaked.
