2

While it can be argued that I should be using some other sort of library systems for my DSL the fact remains that it works on Linux with bash but fails on OpenBSD in both ksh and bash.

I have a shell script that loads environment variables (via source) from other shell files. Once all the files are sourced I then run a standalone app that uses the environment to produce a report.

I'm pretty sure that I'm running out of available memory for variables... but how do I determine how much remains?

envtmpl is my binary stands for environmental templates. It converts the environment to a golang map[string]string and then templates to produce code.

local$ envtmpl --output=./src/zquery/report_Customers.go report.go.envtmpl
./regen.sh[174]: envtmpl: Argument list too long

The argument list is not too long... it has to be the contents of the environment.

UPDATE: I've run your test to see exactly how much "memory" I have available. Seems like I have more than I thought and now I have to reconsider what this error message means Argument list too long when there are two simple arguments. So the question is where is this error really coming from and what does it mean?

obsd66:~/dev/zurvita/zquery$export var=1234578901234456789012345678901234567890123456789012345789012344567890123456789012345678901234567890
obsd66:~$while (( i++ < 10000 )) ;do  var=$var$var ; print "i=$i\t" ${#var} ; done
i=1      200
i=2      400
i=3      800
i=4      1600
i=5      3200
i=6      6400
i=7      12800
i=8      25600
i=9      51200
i=10     102400
i=11     204800
i=12     409600
i=13     819200
i=14     1638400
i=15     3276800
i=16     6553600
i=17     13107200
i=18     26214400
i=19     52428800
i=20     104857600
i=21     209715200
i=22     419430400
ksh: internal error: unable to allocate memory
obsd66:~$echo $KSH_VERSION
@(#)PD KSH v5.2.14 99/07/13.2

UPDATE: I replaced my command envtmpl... with exactly ls -l and I received the same error only referencing ls. So it's not environment memory and it's not variable expansion.

UPDATE: I tried ls instead of ls -l with the same results.

UPDATE: getconf ARG_MAX in openbsd produces a number 2x that of my linux machine. Works in linux but not openbsd.

Richard
  • 10,122
  • 10
  • 42
  • 61
  • 3
    what is `envtmpl`? regardless, the error isn't so much an indicator of running out of memory as it is an issue of some command being fed way more arguments that it was designed to handle ... which comes back to 'what is `envtmpl`?' and which command is generating the error message? – markp-fuso Jan 06 '20 at 17:14
  • See https://stackoverflow.com/questions/19354870/bash-command-line-and-input-limit . `man ksh` may have section near the bottom label `LIMITS` which would contain `ksh` specific info. Good luck. – shellter Jan 06 '20 at 18:43
  • AND see my [incomplete test for environment size](https://stackoverflow.com/a/14372188/620097) . If you need an exact number, you'll have to add something that increments that 2nd-to-last loop in smaller increments. Good luck. – shellter Jan 06 '20 at 19:56
  • 1
    `ls -l` still has one argument (two, actually, since the command name itself is the "zeroth" argument for this purpose), which if you've filled the environment with a *lot* of data from your sourced files, is still enough to exceed the env+args limit. – chepner Jan 06 '20 at 21:31
  • 1
    Also, it's not the *number* of arguments or environment variables that matters, it's the combined *length* of all the arguments and the `...=...` strings in the environment. – chepner Jan 06 '20 at 21:36
  • 1
    @markp, you might withdraw the comment (after reviewing the linked duplicates) -- this really is an OS-level per-process limitation, not specific to `envtmpl` in any way; the error itself comes from the `execve` syscall, when the combined command-line and argument length is too large to fit in the relevant region. – Charles Duffy Jan 06 '20 at 21:36
  • I really appreciate the help, however, I disagree with the conclusion that this is a duplicate as the opinion is void of facts or repo. When I ran the test I had plenty of space and when I compared the ARG_MAX I had plenty still. In fact the actual evidence suggests the opposite. – Richard Jan 06 '20 at 22:06

0 Answers0