No need for Object.keys (though that would be fine for getting an array of the object keys to check) or map, just a simple loop where you remember the object with the highest number you've seen so far:
let found = null;
// Loop through the keys of the object
for (const key in obj) {
    // Get the value for this key
    const entry = obj[key];
    // If this is an own property of the object and either A) We
    // don't have any "found" object yet or the one we do have has
    // a lower `num` than this one, remember this one
    if (Object.hasOwn(obj, key) && (!found || found.num < entry.num)) {
        found = entry;
    }
}
Live Example:
const obj = {
    first:{
        num:10
    },
    second: {
        num:5
    },
    third: {
        num: 15
    }
};
let found = null;
for (const key in obj) {
    const entry = obj[key];
    if (Object.hasOwn(obj, key) && (!found || found.num < entry.num)) {
        found = entry;
    }
}
console.log(found);
 
 
If you also need to remember the property name, just do that at the same time you assign to found.
That uses the newish Object.hasOwn function. If you need to support implementations that don't have it, use a curated polyfill or just:
if (!Object.hasOwn) {
    Object.defineProperty(Object, "hasOwn", {
        value: Function.prototype.call.bind(Object.prototype.hasOwnProperty),
        writable: true,
        configurable: true,
    });
}