So, when we have a simple constructor:
function Vec(x, y) {
    this.x = x;
    this.y = y;
}
And a factory analogue:
function VecFactory(x, y) {
    return {
        x: x,
        y: y
    }
}
The performance is comparable:
100000000 constructors: 1874 ms
100000000 factory calls: 1782 ms
But when we add a prototype:
function VecFactory(x, y) {
    let result = {
        x: x,
        y: y
    }
    Object.setPrototypeOf(result, Vec.prototype);
    return result;
}
The performance drops drastically
100000000 factory calls: 13251 ms
Assigning to __proto__ makes it worse, cloning all prototype methods into every object even worse still.
So, why does this happen? Is there a way to improve factory performance?
UPD: using Object.create instead of Object.setPrototypeOf, as Bergi suggests, drops factory call time to around 8700 ms.