As tehtmi has pointed out, function Vector3:new(x,y,z) is equivalent to function Vector3.new(self,x,y,z). The call Vector3.new(1,2,3) thus sets self = 1, x = 2, y = 3 and z = nil; 2 is printed.
There are two ways to fix this:
- Fix your constructor to be simply Vector3.new, removing theselfparameter.
- Fix your call to be Vector3:new, passingself = Vector3. This obsoletes theVector3global variable access in the constructor;setmetatable(instance, Vector3)can then be replaced withsetmetatable(instance, self).
If you rewrite the constructor slightly, you get the standard "class" template outlined in the "Programming in Lua" book:
function Vector3:new(x,y,z)
  local instance = 
  {
    x = x or 0,
    y = y or 0,
    z = z or 0
  }
  setmetatable(instance, self)
  self.__index = self -- note the addition of this line
  return instance
end
You can then remove the Vector3.__index = Vector3 line since this will be done in the constructor every time you call Vector3:new(...) (arguably questionable performance-wise). The advantage of this approach is how it deals with inheritance: Inheritance is implemented simply as instantiation; you do not need to take care of writing a new constructor. Say you want a class Scale to extend your Vector3 class:
Scale = Vector3:new() -- a scale is a vector ("inherits")
function Scale:scale(vec) -- scale a vector (componentwise multiplication)
    return Vector3:new(self.x * vec.x, self.y * vec.y, self.z * vec.z)
end
Usage:
-- Instantiation:
-- This wouldn't work if you used `.`
-- since then `Scale.new` would be just `Vector3.new`
-- which sets `Vector3` and not `Scale` as the metatable,
-- but since the `:` passes `Scale`/`Vector3` as `self`, everything is fine.
local s = Scale:new(1, 2, 3)
local v = s:scale(Vector3.new(4, 5, 6))
print(v.x, v.y, v.z) -- 4, 10, 18
You ultimately have to decide whether you prefer constructors looking up their class through an upvalue or global variable or having it passed as the self parameter.
As a rule of thumb, if you define a function using :, you should probably call it using :. Such functions are often called "methods". See difference between . and : in Lua.
Besides, I would rename printX to getX or remove it entirely (since it is currently redundant with just directly accessing x).
Note also that default values are often implemented by sticking them into the class table (in your case Vector3.x, Vector3.y, Vector3.z = 0, 0, 0 would allow you to eliminate the or 0's in the constructor).