First, keep track of how you're returning and printing items. You're attempting to use both alert and console.log for this job when you should be only using one. In addition, you're attempting to console.log the return value of a function that doesn't return anything. For my example below, I chose to use console.log and kept the function returning nothing. Below, I fixed this first problem:
function myFunc() {
    this.a = "aaa";
    return {h: function () {console.log(this.a);}}
}
var f = new myFunc();
f.h.apply(f);
Now onto the real problem. this.a returns undefined because this is bound to different two different things (first an instance of myFunc, and then f) the two times you use it in your code. You can use console.log(this) to verify this.
To solve this problem, bind a variable (let's call it self) to this so that you can reference it later. In JavaScript, inner functions have access to variables bound in outer functions. (This is called closure.) This allows us to use self in the following way:
function myFunc() {
    var self = this;
    this.a = "aaa";
    return {h: function() {console.log(self.a);}}
}
var f = new myFunc();
// f.h.apply(f); // replaced by next line; see first comment
f.h();
This prints the line aaa to the console.