this refers to the current object on which the function is called. When you called test1, you would have done like this
myTest.test1()
Now, test1 is called on myTest object. That is why this refers to myTest in the first case. 
In the second case, you would have executed like this
myTest.test1()()
myTest.test1() returns a function and you are invoking without any current object. In that case, JavaScript will make sure that this will refer the global object (window in this case).
Note: But, in strict mode, this will be undefined, in the second case. You can confirm that like this
var myTest = {};
myTest.test1 = function() {
    return function test2() {
        console.log("this is global", this === window);
        console.log("this is undefined", this === undefined);
        return this;
    }
}
myTest.test1()();
Output will be
this is global true
this is undefined false
But if you include use strict like this
"use strict";
var myTest = {};
myTest.test1 = function() {
    return function test2() {
        console.log("this is global", this === window);
        console.log("this is undefined", this === undefined);
        return this;
    }
}
myTest.test1()();
Output will be
this is global false
this is undefined true