TypeScript really doesn't want to disallow any from matching a type, since that's the whole point of any. You might want to rethink any code which relies on rejecting any, so tread lightly.
That being said, you can use conditional types to build a detector for any which can then be used to disallow an any variable.
Here's the detector:
type IfAny<T, Y, N> = 0 extends (1 & T) ? Y : N;
The type constraint 0 extends 1 is not satisfied (0 is not assignable to 1), so it should be impossible for 0 extends (1 & T) to be satisfied either, since (1 & T) should be even narrower than 1. However, when T is any, it reduces 0 extends (1 & any) to 0 extends any, which is satisfied. That's because any is intentionally unsound and acts as both a supertype and subtype of almost every other type. Therefore, IfAny<T, Y, N> checks if T is any. If so, it returns Y. If not, it returns N. Let's see it work:
type IsAny<T> = IfAny<T, true, false>
const yes: IsAny<any> = true;
const no: IsAny<string> = false;
Recall that I said any matches almost every other type. The only type that doesn't match any is never:
declare const any: any;
const never: never = any; // error, any is not assignable to never
We need that fact too, in order to reject any parameters. Let's change the first signature of f() from
function f(key: undefined): void;
to
function f<K extends IfAny<K, never, undefined>>(key: K): void;
We've made the key a generic type K that is constrained to IfAny<K, never, undefined>. If K is not any, then that constraint is just undefined, so K can only be undefined as desired. If K is any, then that constraint becomes never, and since any does not match never, it will fail to meet the constraint.
When we use the above signature, you see the following behavior:
f(undefined); // still works
f(""); // still error, "" is not assignable to undefined
declare var x: any;
f(x); // now error, any is not assignable to never
which is what you wanted.
Playground link to code