Linux environment variables behind the scenes
In this answer I want to explain how environment variable work in a bit more detail under Linux to help demystify them a bit.
It is likely that concepts will be similar in other operating systems, but I am not familiar with them.
Where are environment variables stored inside a program?
Main question: Where is the environment string actual stored?.
When you ask your operating system to launch a program, besides the path of the executable, you also tell it some extra information about how you'd like to launch it, notably:
- what the program's environment variables?
- what the program's command line arguments?
While starting a program, the operating system will store this extra information in the program's RAM memory itself, just like any other variable programmers have code into it, so the program can basically just access them like global variables.
Also, each running program owns their own version of its environment variables, which could be different from that of any other program.
In Linux, environment variables are stored at the top of program's virtual memory, near the command line arguments:
You could also check this by just printing out the address of different types of variables in a C program along the lines of:
#include <stdio.h>
int main(int argc, char **argv, char **env)
{
printf("argv holds %p\n", argv);
printf("argv[0] holds %p\n", argv[0]);
printf("env holds %p\n", env);
printf("env[0] holds %p\n", env[0]);
return 0;
}
which would allow you to reconstruct the memory layout yourself.
How are environment variable set by the actual system call that starts a program?
A system call in Linux is the name of the interfaces that programs use to talk to the operating system and ask it to do things like writing to stdout and launch a new process.
The system call that starts a new process is called execv. By looking at its manual page:
man execv
we can see a few variants of the C wrappers that can be used to call the actual system call:
int execl(const char *pathname, const char *arg, ...
/*, (char *) NULL */);
int execlp(const char *file, const char *arg, ...
/*, (char *) NULL */);
int execle(const char *pathname, const char *arg, ...
/*, (char *) NULL, char *const envp[] */);
int execv(const char *pathname, char *const argv[]);
int execvp(const char *file, char *const argv[]);
int execvpe(const char *file, char *const argv[], char *const envp[]);
Further down it explains how the C interfaces decide the environment variables of the new process:
e - execle(), execvpe()
The environment of the new process image is specified via the argument envp. The envp argument is an array of pointers to null-terminated
strings and must be terminated by a null pointer.
All other exec() functions (which do not include 'e' in the suffix) take the environment for the new process image from the external vari‐
able environ in the calling process.
At: https://stackoverflow.com/questions/2050961/is-argv0-name-of-executable-an-accepted-standard-or-just-a-common-conventi/42291142#42291142 for example I give an example of using the execve variant to set environment variables:
#define _XOPEN_SOURCE 700
#include <unistd.h>
int main(void) {
char argv[] = {"yada yada", NULL};
char envp[] = {NULL};
execve("b.out", argv, envp);
}
I think the final syscall just takes the "array of pointers to null terminated strings" as input but it would be good to confirm.
How to set environment variables from your shell for one command?
This is highly shell dependent and was well covered at: https://superuser.com/a/284361/128124
It is worth highlighting however that under the hood, the shell is just converting the command string to an underlying execv call which does the actual work of starting the program with the desired environment variables.
How to set environment variables from configuration files to be used for all commands
See e.g.: https://stackoverflow.com/questions/532155/linux-where-are-environment-variables-stored
PATH environment variable implementation
Now let's turn our attention to the PATH environment variable in particular.
Implementation wise, some of the C wrappers mentioned in man execv implement the PATH searching. I believe this is done in userland, and that the Linux kernel takes the final resolved absolute path of the executable found only.
man execv mentions:
p - execlp(), execvp(), execvpe()
These functions duplicate the actions of the shell in searching for an executable file if the specified filename does not contain a slash (/) character. The file is sought in the colon-separated list of directory pathnames specified in the PATH environment variable. If this variable isn't defined, the path list defaults to a list that includes the directories returned by confstr(_CS_PATH) (which typically re‐ turns the value "/bin:/usr/bin") and possibly also the current working directory; see NOTES for further details.
execvpe() searches for the program using the value of PATH from the caller's environment, not from the envp argument.
If the specified filename includes a slash character, then PATH is ignored, and the file at the specified pathname is executed.
In addition, certain errors are treated specially.
If permission is denied for a file (the attempted execve(2) failed with the error EACCES), these functions will continue searching the rest of the search path. If no other file is found, however, they will return with errno set to EACCES.
If the header of a file isn't recognized (the attempted execve(2) failed with the error ENOEXEC), these functions will execute the shell (/bin/sh) with the path of the file as its first argument. (If this attempt fails, no further searching is done.)
All other exec() functions (which do not include 'p' in the suffix) take as their first argument a (relative or absolute) pathname that identifies the program to be executed.
PATH environment variable standardization
One notable standardization effort that covers the PATH environment variable is POSIX, which is followed to a large extent by Linux and Mac OS but not Windows.
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08 covers the variable itself:
PATH
This variable shall represent the sequence of path prefixes that certain functions and utilities apply in searching for an executable file known only by a filename. The prefixes shall be separated by a <colon> ( ':' ). When a non-zero-length prefix is applied to this filename, a <slash> shall be inserted between the prefix and the filename if the prefix did not end in <slash>. A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent <colon> characters ( "::" ), as an initial <colon> preceding the rest of the list, or as a trailing <colon> following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH. The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found. If the pathname being sought contains a <slash>, the search through the path prefixes shall not be performed. If the pathname begins with a <slash>, the specified path is resolved (see Pathname Resolution). If PATH is unset or is set to null, the path search is implementation-defined.
Since <colon> is a separator in this context, directory names that might be used in PATH should not include a <colon> character.
There is also a page about the execv family of functions: https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html#tag_16_111 which appears to contain the same contents as man execv, talking both about PATH sarch and environment variable setting.
+R).
