Lua table.insert() is overwriting all indices in my table

Question

The table.insert() function is overwriting all indices in my table. I am trying to append tokens to the table to pass on to a parser, but it isn't working. Here are some code snippets:

Tokenizing, where the problem is happening:

function Lexer:Tokenize()
  local Tokens = {}

  while self.CC ~= nil do
    if string.find(" \t", self.CC) then
      self:Advance()

    -- Numbers
    elseif string.find(C.DIGITS, self.CC) then
      table.insert(Tokens, self:MakeNumber())

    -- Operators
    elseif self.CC == "+" then
      table.insert(Tokens, TOK:new(C.TT_PLUS, nil, self.Pos))
      self:Advance()
    elseif self.CC == "-" then
      table.insert(Tokens, TOK:new(C.TT_MINUS, nil, self.Pos))
      self:Advance()
    elseif self.CC == "*" then
      table.insert(Tokens, TOK:new(C.TT_MUL, nil, self.Pos))
      self:Advance()
    elseif self.CC == "/" then
      table.insert(Tokens, TOK:new(C.TT_DIV, nil, self.Pos))
      self:Advance()
    elseif self.CC == "(" then
      table.insert(Tokens, TOK:new(C.TT_LPAREN, nil, self.Pos))
      self:Advance()
    elseif self.CC == ")" then
      table.insert(Tokens, TOK:new(C.TT_RPAREN, nil, self.Pos))
      self:Advance()

    -- Errors
    else
      local PosStart = self.Pos:Copy()
      local Char = self.CC
      self:Advance()
      return {}, E.IllegalCharError:new(string.format("'%s'",Char, PosStart, self.Pos))
    end
  end

  for k, v in pairs(Tokens) do
    print(k, v:repr())
  end

  table.insert(Tokens, TOK:new(C.TT_EOF, nil, self.Pos))
  return Tokens, nil
end

And here is the code for the token class. Token:repr() is just a way to make the string look nice.


local C = require("constants")

local Token = {
  Type = "",
  Value = nil,
  PosStart = nil,
  PosEnd = nil
}

-- Token initializer
function Token:new(Type, Value, PosStart, PosEnd)
  Value = Value or nil
  PosStart = PosStart or nil
  PosEnd = PosEnd or nil
  setmetatable({}, Token)

  self.Type = Type
  self.Value = Value
  if PosStart then
    self.PosStart = PosStart:Copy()
    self.PosEnd = PosStart:Copy()
  end
  if PosEnd then
    self.PosEnd = PosEnd:Copy()
  end

  return self
end

function Token:repr()
  if self.Value ~= nil then
    return string.format("%s: %s", self.Type, self.Value)
  end
  return string.format("%s", self.Type)
end

return Token

Help appreciated :)

Forgot to mention that, when I input "1 + 2" into this, the output is: 1 integer: 2 2 integer: 2 3 integer: 2. — TheTrueBear, Aug 07 '22 at 18:38

Luatic · Accepted Answer · 2022-08-08T17:37:43.287

The problem lies in your Token constructor:

-- Token initializer
function Token:new(Type, Value, PosStart, PosEnd)
  Value = Value or nil
  PosStart = PosStart or nil
  PosEnd = PosEnd or nil
  setmetatable({}, Token)

  self.Type = Type
  self.Value = Value
  if PosStart then
    self.PosStart = PosStart:Copy()
    self.PosEnd = PosStart:Copy()
  end
  if PosEnd then
    self.PosEnd = PosEnd:Copy()
  end

  return self
end

You're assigning to self, but what is self here when you call this constructor as Token:new(...)? It is the "class table" Token. The line setmetatable({}, Token) has no effect; the newly created table is discarded immediately after since you're not assigning it to any variable.

Your constructor does not require passing Token since you already have & use Token as an upvalue. If you're not interested in having further classes inherit from Token, and since you're trying to keep the constructor simple, I'd recommend simply getting rid of the unnecessary self argument (you were probably copying some Lua OOP pattern, where this is used to allow implementing inheritance later on):

function Token.new(Type, Value, PosStart, PosEnd)

note the . instead of : to not add an implicit self argument. You'll also have to change your calls from TOK:new(...) to just TOK.new(...) (notice again swapping : for . since there is no need to pass the token class to one of its functions).

self has to be the newly created instance:

local self = setmetatable({}, Token)

everything should work if you make these changes. table.insert is not the culprit here.

So, for the :repr() method, what I want to do is .repr() instead, and you pass in the new object, right? — TheTrueBear, Aug 07 '22 at 18:59
I tested it, and it works. Thanks so much! I don't use OOP that much in Lua. — TheTrueBear, Aug 07 '22 at 19:02
@TheTrueBear don't worry, your `repr` method is all fine and idiomatic. My points only apply to the constructor; methods *need* a `self` instance variable passed. — Luatic, Aug 07 '22 at 19:10

Lua table.insert() is overwriting all indices in my table

1 Answers1