The value of this is set by the Javascript interpreter based on how a function is called, NOT on how it is declared.
In this case, you are calling the function as just a normal function so this will be set to either the global object (which is window in a browser) or to undefined (in strict mode).
This part of your declaration:
(function() {
    console.log(this.name); //returns nothing 
}())
is just a normal IIFE function call so thus this is set to window or undefined (if in strict mode).
See this answer for a complete description of the five different ways that this can be controlled by the calling code.
In addition, your myFunc property ends up undefined because you have an IIFE as the value, but that IIFE does not return anything so the return value is undefined thus the resulting value of the myFunc property is undefined.
Since you seem to be having a hard time understand how your IIFE works, your code here:
var myObj = {
  name : 'luke',
  age : '24',
  myFunc: (function() {
    console.log(this.name); //returns nothing 
  }())
};
evaluates the same as this:
function aFunc() {
    console.log(this.name);
}
var myObj = {
  name : 'luke',
  age : '24',
  myFunc: aFunc()
};
And, from this, you should be able to see that aFunc() is a normal function call which causes Javascript to set this inside that function to either window or undefined (if in strict mode).