HttpClient is only available in 4.5 NET and above and Unity does not use that version. Unity uses about 3.5 .NET version.
If you are using Unity 5.3, UnityWebRequest.Delete can be used to make a Delete request. It can be found in the Experimental.Networking namespace. If you are using Unity 5.4 and above,UnityWebRequestcan be found in the UnityEngine.Networking; namespace.
Full working example:
IEnumerator makeRequest(string url)
{
    UnityWebRequest delReq = UnityWebRequest.Delete(url);
    yield return delReq.Send();
    if (delReq.isError)
    {
        Debug.Log("Error: " + delReq.error);
    }
    else
    {
        Debug.Log("Received " + delReq.downloadHandler.text);
    }
}
Usage:
StartCoroutine(makeRequest("http://www.example.com/whatever"));
Make sure to include using UnityEngine.Networking. You can find complete examples with it here.
EDIT (UPDATE)
Unity now supports .NET 4.5 so you can now use HttpClient if you wish. See this post for how to enable it.
After enabling it, 
Go to <UnityInstallationDirectory>\Editor\Data\MonoBleedingEdge\lib\mono\4.5 or for example, C:\Program Files\Unity\Editor\Data\MonoBleedingEdge\lib\mono\4.5 on my computer.
Once in this directory, copy System.Net.Http.dll to your <ProjectName>\Assets directory and you should be able to use HttpClient after importing the System.Net.Http namespace. If there are some other error about missing dependencies, you can get the dlls from this path too and copy them to your <ProjectName>\Assets directory too.