Import a module relative to a file path

Question

I'm working on a macro engine that transforms YAML files. These YAML files contain paths to Python modules that I am importing using importlib. I would like for end users to be able to specify relative paths beginning with ., and for these paths to be resolved relative to the YAML file. (This way, a user could easily ship the YAML file and a related module in a directory or zip file.)

I would prefer not to modify sys.path if possible, but this is not a hard requirement (I can use a context manager to patch/unpatch it).

I know how to use importlib.import_module(name, package) to import name relative to a dotted path package. But here, I have an OS file path to the YAML file, which is not a Python module. Can this be done?

Example:

My script is at ~/bin/macroengine.py
The YAML file is at ~/example/source.yaml
The external module is at ~/example/myModule.py

I would like for source.yaml to reference the external module as .myModule.

Can you gives examples of paths you struggling on please? I'm not sure to understand what you means by "an OS file path to the YAML file". — Arount, Dec 28 '17 at 16:38

Arount · Accepted Answer · 2017-12-29T10:18:32.793

Here is the filesystem paths I used to test:

/tmp/stack/ymport/content.yaml:

afile: .foo.bar.baz.afile
amodule: .egg.bacon

/tmp/stack/ymport/foo/bar/baz/afile.py:
```
variable = 'A FILE'
```
/tmp/stack/ymport/egg/bacon/__init__.py:
```
variable = 'A MODULE'
```

Python script:

import os
import yaml
from importlib.machinery import SourceFileLoader


def ymport(module_name, base_dir=None):
    '''
    Import module from relative path.
        module_name    Name / path-string of the module (foo.bar.baz)
        base_dir       Base directory to find module (default './')

    If module can not be found as file (foo/bar/baz.py) it will try to import it
    as module (foo/bar/baz/__init__.py).

    Returns module instance
    '''

    if base_dir is None:
        base_dir = './'

    base_path = relative_to_absolute(module_to_os_path(module_name), base_dir)
    file_path = '{}.py'.format(base_path)


    try:
        return SourceFileLoader(module_name, file_path).load_module()
    # If more obvious path didn't works, try to import path as module (__init__.py)
    except FileNotFoundError:
        module_path = '{}/__init__.py'.format(base_path)
        try:
            return SourceFileLoader(module_name, module_path).load_module()
        except FileNotFoundError:
            # Make obvious we tried 2 differents paths
            raise FileNotFoundError("No such files or directories '{}', '{}'".format(
                file_path, module_path
            ))


def module_to_os_path(module_name):
    '''
    Parse module path (foo.bar.baz) into filesystem path (foo/bar/baz)
    '''
    if module_name.startswith('.'):
        module_name = module_name[1:]

    return module_name.replace('.', os.sep)


def relative_to_absolute(path, base):
    return os.path.join(base, path)


# Let's try it
with open('/tmp/stack/ymport/content.yaml') as fh:
    base_path = os.path.dirname(fh.name)
    data = yaml.load(fh.read())

    for name, path in data.items():
        module = ymport(path, base_path)
        print(module.variable)

The output:

A FILE
A MODULE

Import from absolute filesystem path as reference.

Some notes:

Done for Python3.3.x
It allow you to load both modules and files (foo.py vs foo/__init__.py).
You may needs to update it according to specifics needs, but the basics are here.

I'd like to be able to import foo as `.foopackage.foo` rather than as `.foopackage/foo.py`. Unfortunately, this is a hard requirement because the system already supports absolute package paths in the usual dotted-path style. Also, it looks like some of the functions used here aren't available in Python 3.3. — Thom Smith, Dec 28 '17 at 18:51
Done - let me know if it fit your need (at least if you have enough to handle it) — Arount, Dec 29 '17 at 10:15

Import a module relative to a file path

1 Answers1