If the response is parsed as JSON, the jqXHR object will have a responseJSON property.
$.ajax(
    // ...
).done(function(data, textStatus, jqXHR) {
    if (jqXHR.responseJSON) {
        // handle JSON
    } else {
        // handle html
    }
}).fail(function(jqXHR, textStatus, errorThrown) {
    if (jqXHR.responseJSON) {
        // handle JSON
    else {
        // handle html
    }
})
From the jQuery.ajax documentation:
If json is specified, the response is parsed using jQuery.parseJSON before being passed, as an object, to the success handler. The parsed JSON object is made available through the responseJSON property of the jqXHR object.