python setuptools install_requires is ignored when overriding cmdclass

Question

I have a setup.py that looks like this:

from setuptools import setup
from subprocess import call
from setuptools.command.install import install

class MyInstall(install):
    def run(self):
        call(["pip install -r requirements.txt --no-clean"], shell=True)
        install.run(self)

setup(
    author='Attila Zseder',
    version='0.1',
    name='entity_extractor',
    packages=['...'],
    install_requires=['DAWG', 'mrjob', 'cchardet'],
    package_dir={'': 'modules'},
    scripts=['...'],
    cmdclass={'install': MyInstall},
)

I need MyInstall because I want to install some libraries from github and I didn't want to use dependency_links option, because it's discouraged (for example here), so I can do this with requirements.txt.

When I install this package with pip, everything is working fine, but for some reasons I have to solve this in a way that it also works with pure python setup.py install. And it doesn't.

When overriding cmdclass in setup() with my own class, install_requires seems to be ignored. As soon as I comment out that line, those packages are being installed.

I know that install_requires is not supported for example in distutils (if I remember well), but it is in setuptools. And then cmdclass wouldn't have any effect on install_requires.

I googled this problem for hours, found a lot of kind of related answers on stackoverflow, but not for this particular problem.

With putting every needed package to requirements.txt, everything's working fine, but I would like to understand why this is happening. Thanks!

score 16 · Accepted Answer · answered Mar 04 '14 at 17:54

16

The same problem just happened to me. It somehow seems like something triggers setuptools to do an 'old-style install' with distutils, which indeed does not support install_requires.

You call install.run(self) which calls run(self) in setuptools/setuptools/command/install.py, line 51-74

https://bitbucket.org/pypa/setuptools/src/8e8c50925f18eafb7e66fe020aa91a85b9a4b122/setuptools/command/install.py?at=default

def run(self):
    # Explicit request for old-style install?  Just do it
    if self.old_and_unmanageable or self.single_version_externally_managed:
        return _install.run(self)

    # Attempt to detect whether we were called from setup() or by another
    # command.  If we were called by setup(), our caller will be the
    # 'run_command' method in 'distutils.dist', and *its* caller will be
    # the 'run_commands' method.  If we were called any other way, our
    # immediate caller *might* be 'run_command', but it won't have been
    # called by 'run_commands'.  This is slightly kludgy, but seems to
    # work.
    #
    caller = sys._getframe(2)
    caller_module = caller.f_globals.get('__name__','')
    caller_name = caller.f_code.co_name

    if caller_module != 'distutils.dist' or caller_name!='run_commands':
        # We weren't called from the command line or setup(), so we
        # should run in backward-compatibility mode to support bdist_*
        # commands.
        _install.run(self)
    else:
        self.do_egg_install()

I'm not sure whether this behaviour is intended, but replacing

install.run(self)

with

install.do_egg_install()

should solve your problem. At least it works for me, but I would also appreciate a more detailed answer. Thanks!

answered Mar 04 '14 at 17:54

KEgg

176
1
2

It did solve my problem, thank you! But yes, I would also appreciate a detailed answer about the reasons. – zseder Mar 06 '14 at 10:33
1

Interesting observation for anyone wondering: As explained [here](http://www.niteoweb.com/blog/setuptools-run-custom-code-during-install), `setuptools` commands subclass [`distutils.cmd.Command`](https://bitbucket.org/carljm/python-distutils/src/48c42eeaee4410d76675b637bcd401b8919ff19a/cmd.py?at=default#cl-14), which is an old-style Python `class`. For this reason, `super` cannot be used, and the parent class has to be referenced directly. – 0 _ Mar 27 '15 at 10:25
Shouldn't this be `install.do_egg_install(self)`? – Jonathon Reinhart Aug 06 '16 at 04:46
1

I wish I could upvote this answer more than once. Also, when using python3, calling `super().do_egg_install()` worked fine – dusktreader Oct 10 '16 at 23:12
@JonathonReinhart, I think so. I pasted the code and faced a syntax error I ever met long back to the days I started learning python. – John Wang Oct 22 '16 at 14:40
@dusktreader your python3 fix doesn't seem to work for python 3.6.0. – Roman Mar 22 '17 at 10:09
1

Using this solution I got `ERROR: Failed building wheel` messages when installing my package from PyPi. My conclusion is that `cmdclass` should not be relied on for production modules. It's too poorly documented and has too many edge cases where things unexpectedly fail. – Quantum7 Jun 27 '19 at 12:24
this does not work for python 3.6; it creates the egg folder under the wheel folder while it is expected to be under `build/bdist.linux-x86_64/` – dada Jul 01 '20 at 08:00

score 12 · Answer 2 · edited May 23 '17 at 11:54

According to https://stackoverflow.com/a/20196065 a more correct way to do this may be to override bdist_egg command.

You could try:

from setuptools.command.bdist_egg import bdist_egg as _bdist_egg

class bdist_egg(_bdist_egg):
    def run(self):
        call(["pip install -r requirements.txt --no-clean"], shell=True)
        _bdist_egg.run(self)

...

setup(...
    cmdclass={'bdist_egg': bdist_egg},  # override bdist_egg
)

It worked for me and install_requireis no more ignored. Nevertheless, I still don't understand why most people seem to override cmdclass install and do not complain about install_require being ignored.

Perhaps they are only using `pip install` and never `python setup.py install`, or when using the latter they do so in a `virtualenv` already populated with dependencies. — 0 _, Mar 27 '15 at 10:19

David Vitale · Answer 3 · 2018-01-25T19:29:07.707

2

I know this is an old question, but I ran into a similar problem. The solution I have found fixes this problem for me is very subtle: The install class you're setting in cmd_class must physically be named install. See this answer on a related issue.

Note that I use the class name install for my derived class because that is what python setup.py --help-commands will use.

You also should use self.execute(_func_name, (), msg="msg") in your post_install instead of calling the function directly

So implementing something like this should cause you to avoid the do_egg_install workaround implemented above by KEgg.

from setuptools.command.install import install as _install
...
def _post_install():
    #code here
class install(_install):
    def run(self):
        _install.run(self)
        self.execute(_post_install, (), msg="message here")

edited Jan 25 '18 at 19:29

answered Jan 25 '18 at 18:45

David Vitale

176
2
3

1

Are you sure with that? I can't reproduce and that also doesn't solve the frame check in setuptools' `run` implementation. – funky-future Mar 13 '18 at 20:51
Honestly, I am not sure. `setuptools` is very underdocumented, and often behaves sporadically. I was able to get the desired behavior for my PyPI package with reproducible post-install commands, perhaps in ways I don't understand. You can download, unzip, and then examine the setup.py for yourself: https://pypi.python.org/pypi/arcgis – David Vitale Mar 14 '18 at 16:29

python setuptools install_requires is ignored when overriding cmdclass

3 Answers3

Linked