dev_appserver.py app.yaml produces: ImportError: Importing the multiarray numpy extension module failed

Question

I run this command:

dev_appserver.py app.yaml

and I get an error:

Traceback (most recent call last):
  File "C:\Users\sehrlich\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "C:\Users\sehrlich\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "C:\Users\sehrlich\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "C:\Users\sehrlich\Desktop\ToolKit\Application Projects\Tableau Web Data Connector Improvement\tableauextensions\main.py", line 2, in <module>
    from Get_Data import get_data, build_connection, run_sql
  File "C:\Users\sehrlich\Desktop\ToolKit\Application Projects\Tableau Web Data Connector Improvement\tableauextensions\Get_Data.py", line 1, in <module>
    import numpy as np
  File "C:\Users\sehrlich\Desktop\ToolKit\Application Projects\Tableau Web Data Connector Improvement\tableauextensions\lib\numpy\__init__.py", line 142, in <module>
    from . import add_newdocs
  File "C:\Users\sehrlich\Desktop\ToolKit\Application Projects\Tableau Web Data Connector Improvement\tableauextensions\lib\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "C:\Users\sehrlich\Desktop\ToolKit\Application Projects\Tableau Web Data Connector Improvement\tableauextensions\lib\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "C:\Users\sehrlich\Desktop\ToolKit\Application Projects\Tableau Web Data Connector Improvement\tableauextensions\lib\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "C:\Users\sehrlich\Desktop\ToolKit\Application Projects\Tableau Web Data Connector Improvement\tableauextensions\lib\numpy\core\__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

I have installed and uninstalled NumPy. They app works fine when I run something like

python -m flask run

and it uses NumPy no problem. Can't figure out what the issue is.

Are you [requesting](https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27#using_a_built-in_third-party_library_bundled_with_the_runtime) the GAE provided `numpy` or trying to [vendor in](https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27#copying_a_third-party_library) your own version as the file paths in your tracebacks appear to suggest? (I suspect it's not pure python code, if so it can't be vendored in). — Dan Cornilescu, Sep 26 '18 at 02:54
Hi There @danCornilescu, I am requesting GAE provided numpy. That link is exactly what I was looking for. What is best solution for enabling numpy to be used in my Google App Engine? — shazeltion17, Sep 26 '18 at 14:58

Iñigo · Answer 1 · 2018-09-26T11:06:18.700

As Dan Cornilescu said, GAE Standard can't use libraries with code compiled in C [1] [2]:

[From 1] You can use third-party libraries that are pure Python code with no C extensions

[From 2] The interpreter cannot load Python services with C code; it is a "pure" Python environment.

NumPy is one of those cases, you can see it in their Git Repo [3] and in Wikipedia [4] (Written in: Python, C). Also, check the first answer to this SO question.

Curiously enough, I found a version on NumPy only based on "pure" Python called "TinyNumPy" [5] that you could use in GAE Standard. This are its limitations according to their Git Repo:

ndarray.flat iterator cannot be indexed (it is a generator).

No support for Fortran order.

Support for data types limited to bool, uin8, uint16, uint32, uint64, int8, int16, int32, int64, float32, float64.

Functions that calculate statistics on the data are much slower, since the iteration takes place in Python.

Assigning via slicing is usually pretty fast, but can be slow if the striding is unfortunate.

In a nutshell, either you use GAE Flex or try to avoid NumPy.

score 1 · Answer 2 · answered Sep 26 '18 at 15:10

Since you want to use the GAE-supplied numpy then:

you shouldn't have it installed inside your application code (the traceback indicates it's running from the app's lib dir, where vendored-in libraries go)
you should request it in your app.yaml's libraries section:
```
libraries:
- name: numpy
  version: "1.6.1"
```
you should also have the requested numpy version installed locally on your system (but not in the app dir) so that the development server can use it as numpy is one of the libraries with such requirement, see Using built-in bundled libraries with the local development server:

Many of the built-in libraries provided by the runtime are automatically available to the local development server. However, the following built-in libraries must be installed locally before you can use them with the local development server:

...

numpy

Great Response! This does answer the question correctly! As a side note though, why does google only have this version of Numpy. My application needs to use version 1.9 or later but looks like they do not support it. — shazeltion17, Sep 26 '18 at 16:59
My guess is that it's not exactly trivial to port these packages/versions to make them work decently on GAE, with the sandbox/infra restrictions. So it becomes a matter of resources, priorities, etc. — Dan Cornilescu, Sep 27 '18 at 02:02

dev_appserver.py app.yaml produces: ImportError: Importing the multiarray numpy extension module failed

2 Answers2

Linked