Reading in environment variables from an environment file

Question

I'd like to run in a local environment a Python script which is normally run in a Docker container. The docker-compose.yml specifies an env_file which looks (partially) like the following:

DB_ADDR=rethinkdb
DB_PORT=28015
DB_NAME=ipercron

In order to run this locally, I would like these lines to be converted to

os.environ['DB_ADDR'] = 'rethinkdb'
os.environ['DB_PORT'] = '28015'
os.environ['DB_NAME'] = 'ipercron'

I could write my parser, but I was wondering if there are any existing modules/tools to read in environment variables from configuration files?

Answer 1

I use Python Dotenv Library. Just install the library pip install python-dotenv, create a .env file with your environment variables, and import the environment variables in your code like this:

import os
from dotenv import load_dotenv

load_dotenv()

MY_ENV_VAR = os.getenv('MY_ENV_VAR')

From the .env file:

MY_ENV_VAR="This is my env var content."

This is the way I do when I need to test code outside my docker system and prepare it to return it into docker again.

Answer 2

If your system/environment/workflow supports using shell scripts, you can create a script that wraps these 2 operations:

1. Sourcing the .env file and exporting them as environment variables
   • Using the set -a option where "Each variable or function that is created or modified is given the export attribute and marked for export to the environment of subsequent commands".
2. Calling your Python script/app that has plain os.environ.get code

Sample .env file (config.env):

TYPE=prod
PORT=5000

Sample Python code (test.py):

import os

print(os.environ.get('TYPE'))
print(os.environ.get('PORT'))

Sample bash script (run.sh):

#!/usr/bin/env bash

set -a
source config.env
set +a

python3 test.py

Sample run:

$ tree
.
├── config.env
├── run.sh
└── test.py

$ echo $TYPE

$ echo $PORT

$ python3 test.py
None
None

$ ./run.sh 
prod
5000

When you run the Python script directly (python3 test.py) without source-ing the .env file, all the environ.get calls return None.

But, when you wrap it in a shell script that first loads the .env file into environment variables, and then runs the Python script afterward, the Python script should now be able to read the environment variables correctly. It also ensures that the exported env vars only exist as part of the execution of your Python app/script.

As compared with the other popular answer, this doesn't need any external Python libraries.

Answer 3

This could also work for you:

env_vars = [] # or dict {}
with open(env_file) as f:
    for line in f:
        if line.startswith('#') or not line.strip():
            continue
        # if 'export' not in line:
        #     continue
        # Remove leading `export `, if you have those
        # then, split name / value pair
        # key, value = line.replace('export ', '', 1).strip().split('=', 1)
        key, value = line.strip().split('=', 1)
        # os.environ[key] = value  # Load to local environ
        # env_vars[key] = value # Save to a dict, initialized env_vars = {}
        env_vars.append({'name': key, 'value': value}) # Save to a list

print(env_vars)

In the comments, you'll find a few different ways to save the env vars and also a few parsing options i.e. to get rid of the leading export keyword. Another way would be to use the python-dotenv library. Cheers.

UPDATE: I setup my own envvar_utils.py to handle conversion from string etc..

"""Utility functions for dealing with env variables and reading variables from env file"""
import os
import logging
import json

BOOLEAN_TYPE = 'boolean'
INT_TYPE = 'int'
FLOAT_TYPE = 'float'
STRING_TYPE = 'str'
LIST_TYPE = 'list'
DICT_TYPE = 'dict'


def get_envvars(env_file='.env', set_environ=True, ignore_not_found_error=False, exclude_override=()):
    """
    Set env vars from a file
    :param env_file:
    :param set_environ:
    :param ignore_not_found_error: ignore not found error
    :param exclude_override: if parameter found in this list, don't overwrite environment
    :return: list of tuples, env vars
    """
    env_vars = []
    try:

        with open(env_file) as f:
            for line in f:
                line = line.replace('\n', '')

                if not line or line.startswith('#'):
                    continue

                # Remove leading `export `
                if line.lower().startswith('export '):
                    key, value = line.replace('export ', '', 1).strip().split('=', 1)
                else:
                    try:
                        key, value = line.strip().split('=', 1)
                    except ValueError:
                        logging.error(f"envar_utils.get_envvars error parsing line: '{line}'")
                        raise

                if set_environ and key not in exclude_override:
                    os.environ[key] = value

                if key in exclude_override:
                    env_vars.append({'name': key, 'value': os.getenv(key)})
                else:
                    env_vars.append({'name': key, 'value': value})
    except FileNotFoundError:
        if not ignore_not_found_error:
            raise

    return env_vars


def create_envvar_file(env_file_path, envvars):
    """
    Writes envvar file using env var dict
    :param env_file_path: str, path to file to write to
    :param envvars: dict, env vars
    :return:
    """
    with open(env_file_path, "w+") as f:
        for key, value in envvars.items():
            f.write("{}={}\n".format(key, value))
    return True


def convert_env_var_flag_to(env_var_name, required_type, default_value):
    """
    Convert env variable string flag values to required_type
    :param env_var_name: str, environment variable name
    :param required_type: str, required type to cast the env var to
    :param default_value: boolean, default value to use if the environment variable is not available
    :return: environment variable value in required type
    """
    env_var_orginal_value = os.getenv(env_var_name, default_value)
    env_var_value = ""
    try:
        if required_type == INT_TYPE:
            env_var_value = int(env_var_orginal_value)
        elif required_type == FLOAT_TYPE:
            env_var_value = float(env_var_orginal_value)
        elif required_type == BOOLEAN_TYPE:
            env_var_value = bool(int(env_var_orginal_value))
        elif required_type == STRING_TYPE:
            env_var_value = str(env_var_orginal_value)
        elif required_type == LIST_TYPE:
            env_var_value = env_var_orginal_value.split(',') if len(env_var_orginal_value) > 0 else default_value
        elif required_type == DICT_TYPE:
            try:
                env_var_value = json.loads(env_var_orginal_value) if env_var_orginal_value else default_value
            except Exception as e:
                logging.error(f"convert_env_var_flag_to: failed loading {env_var_orginal_value} error {e}")
                env_var_value = default_value
        else:
            logging.error("Unrecognized type {} for env var {}".format(required_type, env_var_name))

    except ValueError:
        env_var_value = default_value
        logging.warning("{} is {}".format(env_var_name, env_var_orginal_value))

    return env_var_value

Answer 4

You can use ConfigParser. Sample example can be found here.

But this library expects your key=value data to be present under some [heading]. For example, like:

[mysqld]
user = mysql  # Key with values
pid-file = /var/run/mysqld/mysqld.pid
skip-external-locking
old_passwords = 1
skip-bdb      # Key without value
skip-innodb

Answer 5

Dewald Abrie posted a good solution.

Here's a slight modification that ignores breaklines (\n)

def get_env_data_as_dict(path: str) -> dict:
    with open(path, 'r') as f:
       return dict(tuple(line.replace('\n', '').split('=')) for line
                in f.readlines() if not line.startswith('#'))

print(get_env_data_as_dict('../db.env'))

Answer 6

How about this for a more compact solution:

import os

with open('.docker-compose-env', 'r') as fh:
    vars_dict = dict(
        tuple(line.replace('\n', '').split('='))
        for line in fh.readlines() if not line.startswith('#')
    )

print(vars_dict)
os.environ.update(vars_dict)

Answer 7

Using only python std

import re

envre = re.compile(r'''^([^=]+)\s+?=\s+?(?:[\s"']*)(.+?)(?:[\s"']*)$''')
result = {}
with open('/etc/os-release') as ins:
    for line in ins:
        match = envre.match(line)
        if match is not None:
            result[match.group(1)] = match.group(2)

Answer 8

I think you should leave it for external tools to manage the environment for you.

This way you can easily use secret manager like 1password cli to get the environment variables loaded from encrypted vault like so

op run --env-file=.env -- python your_script.py

Having that said load_dotenv is smart enough not to load .env variables if they are present in the environment but some of the other solutions aren't.

And if you don't have any external tool at your disposal just use 'bash':

set -o allexport; source .env; set +o allexport

Solution taken from: Set environment variables from file of key/value pairs

Answer 9

python-decouple is a good option for these types of files. While it is often used for django applications, it is extensible for arbitrary environment files.

from decouple import Config, RepositoryEnv

env=Config(RepositoryEnv('.env'))

env.get('DB_ADDR') #=> 'rethinkdb'
env.get('DB_PORT') #=> '28015'
env.get('DB_NAME') #=> 'ipercron'

Answer 10

I would not recommend directly reading .env files from within your program. The idea of the config part in 12 factor app is that you read the config from the environment variables -- i.e. not from files.

.env files are just a convenient way to get those variables into the environment (see below). If you start reading the files directly in your code, you're short cutting this step over the environment variables with all consequences and are basically back to square one, reading config from files.

So what you should do instead, use a tool such as dotenv-cli that reads .env files, exports the variables into the environment and runs your app with the temporary modified environment, like so:

$ dotenv yourapp

Answer 11

In situations where using the python-dotenv wasn't possible, I've used something like the following:

import os

def load_env_file(dotenv_path, override=False):
    with open(dotenv_path) as file_obj:
        lines = file_obj.read().splitlines()  # Removes \n from lines

    dotenv_vars = {}
    for line in lines:
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue

        key, value = line.split("=", maxsplit=1)
        dotenv_vars.setdefault(key, value)

    if override:
        os.environ.update(dotenv_vars)
    else:
        for key, value in dotenv_vars.items():
            os.environ.setdefault(key, value)

It reads the given file and parses lines that have the "=" symbol in them. Value before the symbol will be the key, and value after is the value.

Current environment variables with the same keys as in the env file can either be left untouched or overwritten with the override parameter.

Answer 12

I'm not very proud of this, however, in contrast to other answers, it seems to work if one has lines which end with a comment (after data in the same line). Adapted @Tom's answer.

# have to parse manually since dotenv package is not available
def get_env_data_as_dict(dotenv_path):
    result = {}
    with open(dotenv_path) as file_obj:
        lines = file_obj.read().splitlines()  # Removes \n from lines

    for line in lines:
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        if "#" in line:
            line = line.split("#")[0].strip()
        key, value = line.split("=", maxsplit=1)
        result[key] = value
    return result

Answer 13

If you are using a Jupyter Notebook you can use this magic and point to the .env file.

%reload_ext dotenv
%dotenv path/to/env_vars.env

Documentation here.