Interestingly, none of the above answers takes the test examples of the original question into account, because using them, all of the above checks would fail (because of #3):
fe80:0000:0000:0000:0202:b3ff:fe1e:8329
fe80:0:0:0:202:b3ff:fe1e:8329
fe80::202:b3ff:fe1e:8329 (!)
IPv6 representation rules say:
One or more consecutive groups of zero value may be replaced with a single empty group using two consecutive colons (::),1 but the substitution may only be applied once in the address, because multiple occurrences would create an ambiguous representation.
  https://en.wikipedia.org/wiki/IPv6_address#Representation
As Lua patterns do not have support for Alternation, it is not possible to check IPv6 with a single pattern. You may see David M. Syzdek answer on the complexity of IPv6 Regex: https://stackoverflow.com/a/17871737/1895269
Still, a more standards conforming approach is the following improvement of Paul Kulchenko's answer:
function GetIPType(ip)
  local R = {ERROR = 0, IPV4 = 1, IPV6 = 2, STRING = 3}
  if type(ip) ~= "string" then return R.ERROR end
  -- check for format 1.11.111.111 for ipv4
  local chunks = { ip:match("^(%d+)%.(%d+)%.(%d+)%.(%d+)$") }
  if (#chunks == 4) then
    for _,v in pairs(chunks) do
      if tonumber(v) > 255 then return R.STRING end
    end
    return R.IPV4
  end
  -- check for ipv6 format, should be max 8 'chunks' of numbers/letters
  local addr = ip:match("^([a-fA-F0-9:]+)$")
  if addr ~= nil and #addr > 1 then
    -- address part
    local nc, dc = 0, false      -- chunk count, double colon
    for chunk, colons in addr:gmatch("([^:]*)(:*)") do
      if nc > (dc and 7 or 8) then return R.STRING end    -- max allowed chunks
      if #chunk > 0 and tonumber(chunk, 16) > 65535 then
        return R.STRING
      end
      if #colons > 0 then
        -- max consecutive colons allowed: 2
        if #colons > 2 then return R.STRING end
        -- double colon shall appear only once
        if #colons == 2 and dc == true then return R.STRING end
        if #colons == 2 and dc == false then dc = true end
      end
      nc = nc + 1      
    end
    return R.IPV6
  end
  return R.STRING
end
The script to check:
local IPType = {[0] = "Error", "IPv4", "IPv6", "string"}
local ips = {
  "128.1.0.1",    -- ipv4
  "223.255.254.254",  -- ipv4
  "999.12345.0.0001",   -- invalid ipv4
  "1050:0:0:0:5:600:300c:326b",         -- ipv6
  "1050!0!0+0-5@600$300c#326b",         -- string
  "1050:0:0:0:5:600:300c:326babcdef",     -- string
  "1050:0000:0000:0000:0005:0600:300c:326b",  -- ipv6
  "1050:::600:5:1000::",  -- contracted ipv6 (invalid)
  "fe80::202:b3ff:fe1e:8329",   -- shortened ipv6
  "fe80::202:b3ff::fe1e:8329",  -- shortened ipv6 (invalid)
  "fe80:0000:0000:0000:0202:b3ff:fe1e:8329:abcd",  -- too many groups
  "::1",   -- valid IPv6
  "::",  -- valid IPv6
  ":",   -- string
  "129.garbage.9.1",  -- string
  129.10        -- error
}
for k,v in pairs(ips) do
  print(v, IPType[GetIPType(v)])
end
And the output:
128.1.0.1       IPv4
223.255.254.254 IPv4
999.12345.0.0001        string
1050:0:0:0:5:600:300c:326b      IPv6
1050!0!0+0-5@600$300c#326b      string
1050:0:0:0:5:600:300c:326babcdef        string
1050:0000:0000:0000:0005:0600:300c:326b IPv6
1050:::600:5:1000::     string
fe80::202:b3ff:fe1e:8329        IPv6
fe80::202:b3ff::fe1e:8329       string
fe80:0000:0000:0000:0202:b3ff:fe1e:8329:abcd    string
::1     IPv6
::      IPv6
:       string
129.garbage.9.1 string
129.1   Error