var creates a local variable, scoped to the function in which it appears. Any variable that is declared in the global scope becomes a property of the global object (in a browser, window), and any variable referenced in a function which is not declared with var in that function refers to the surrounding scope, quite possibly the global scope.
There is also a new keyword, let, which is coming in the upcoming version of ECMAScript and is already in some browsers, which creates a block-scoped variable.
So, in your first example (I am going to assume this is running in a browser), you create window.a and window.b which are bound to 3 and 2, respectively. Function line declares local variables a and b. When window.a is passed to line, the parameter x is bound to the value 3 (the value of window.a). Local variables a and b are 5 and 4, respectively. These have nothing to do with window.a and window.b. The calculation a*x+b is, thus, 5*3+4, or 19.
The code b = line(a) - b passes window.a to line, calculates the value 19, subtracts the current value of window.b from it (2), resulting in 17, which is then assigned to window.b. The value of 36 comes from line(3) (which is still 19) plus 17 (the new value of window.b).
When you removed the var from the assignment to b in function line, you changed it so that b is no longer a local variable. In that case, all references to b in the function refer to the global value window.b.
In the case of your Riddle #2, where you got 23, after the first call window.b is equal to 15. When line(a) is called a second time, window.b is set back to 4, the a*x+b calculation still gets 19, and then window.b (4) is added again, making 23.
It is important to note that the var keyword declares a variable that is function-scoped, not block-scoped as you might expect from other C-derived languages. For example, in this function:
function scope(array) {
    var a = 7;
    for (var b = 0; b < array.length; ++b) {
        var c = array[b];
    }
    alert(a + b + c);
}
All the variables have scope which extends over the entire function. In particular, the scope of c is not limited to the for loop.
A variable that is not declared in a particular scope doesn't necessarily reference a global variable, however. If it is nested inside another function scope, it will refer to that nested scope. Consider the following example:
var b = 7;
function outer() {
    var b = 42;
    function inner() {
        return b; // refers to b in outer, not global
    }
    return inner();
}
alert(outer()); // 42, not 7