This is not an issue with the ASP.NET Core MVC asp-action Tag Helper - the problem is down to the fact that delete is not a supported method in HTML forms (see the method section here). 
Although different browsers may handle it differently, Chrome just issues a GET request when it sees delete as the HTML form's method. When you remove [HttpDelete] from your Delete action, it defaults to GET (it's as though you'd added [HttpGet]), which is why the GET verb being used by Chrome now hits your Delete action.
In order to fix this, I suggest using the POST verb, which can be triggered using a method of post in your form and adding [HttpPost] to your delete action. Here's what it looks like:
HTML
<a
    class="btn"
    asp-action="Delete"
    asp-controller="Home"
    asp-route-accountKey="@Model.Item1.AccountKey"
    method="post">Delete</a>
C#
[HttpPost]
public void Delete(string accountKey) { }
Using post is preferred over using get for reasons cited by here.