First we have to clarify what new NewExpression and especially NewExpression are. This can be found in Annex A. The most common situation where this rule applies is when you don't pass arguments to the constructor. I.e.
var obj = new F;
where F refers to a function. So this is the rule that lets you omit the parenthesis.
In your example (var h = new f();), you have parenthesis though, i.e. you are passing an empty list of arguments, so this algorithm does not apply. f() is not NewExpression.
Instead this algorithm applies: new MemberExpression Arguments. It is evaluated in pretty much the same way and the algorithm can found in §11.2.2 as well, just after the algorithm you quoted.
With this in mind, lets go through that algorithm step by step:
1. Let ref be the result of evaluating MemberExpression.
In your example, MemberExpression, is f, i.e. it is a variable. The result of the evaluation is a special Reference object. It's not important here what exactly it is. Just know that it holds information about how to actually get the value from the variable.
So now ref refers to that reference.
2. Let constructor be GetValue(ref).
This is were the value of the variable is actually retrieved and constructor will refer to the function that f refers to.
3. Let argList be the result of evaluating Arguments, producing an internal list of argument values (11.2.4).
In your case, Arguments is () and therefore it is an empty list.
4. If Type(constructor) is not Object, throw a TypeError exception.
It is important to know that functions are objects too! So this step will throw the error if primitive values are being used in the new expression.
5. If constructor does not implement the [[Construct]] internal method, throw a TypeError exception.
All functions (and potentially other objects) implement an internal [[Construct]] property which does the actual instantiation of the new object. If the object does not have such a property, it cannot be used as constructor. How it works for functions is defined in §13.2.2.
6. Return the result of calling the [[Construct]] internal method on constructor, providing the list argList as the argument values.
This is were the actual construction happens. [[Construct]] is itself function and is defined in §13.2.2. The method is the same of every function and is responsible for creating a new object, call the function on that new object and return it or whatever the function returns.
Here is an example of what it would look in JavaScript (partly pseudo-code):
[[Construct]] = function(F, argList) {
// Create new object that in inherits from F.prototype or Object.prototype
var proto = F.prototype;
var obj = Object.create(typeof proto === 'object' ? proto : Object.prototype);
// Call F with this set to obj and pass the argument list
var result = F.apply(obj, argList);
// If result is not an object, return the generated object
return typeof result === 'object' ? result : obj;
};