Why does creating this memoryview raise a ValueError only when assigning to a variable?

Question

Pythons memoryview does not support datetime64 or timedelta. Ok. But when I try to create a memoryview of a structured array that includes a datetime64 or timedelta, it appears to work... unless I assign it to a variable!

In [19]: memoryview(zeros(10, dtype=[("A", "m8[s]")]))
Out[19]: <memory at 0x7f1d455d6048>

In [20]: x = memoryview(zeros(10, dtype=[("A", "m8[s]")]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
ValueError: cannot include dtype 'm' in a buffer

In [21]: x = _19

In [22]: x
Out[22]: <memory at 0x7f1d455d6048>

This seriously challenges my understanding of the way Python fundamentally works. How can f() and x = f() be different, considering that (1) IPythons REPL assigns the output to _19 anyway, and (2) the function/class memoryview has no way of knowing what the caller is going to do with its output?

I am running the code on Python 3.4.1, numpy 1.10.0.dev+fbcc24f, on Linux 2.6.32-431.23.3.el6.x86_64, Scientific Linux release 6.6.

EDIT

On Python 3.5, numpy 1.10.4, I get:

In [50]: memoryview(numpy.zeros(10, dtype=[("A", "m8[s]")]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
ValueError: cannot include dtype 'm' in a buffer

During handling of the above exception, another exception occurred:

SystemError                               Traceback (most recent call last)
<ipython-input-50-5d5ac6c085fa> in <module>()
----> 1 memoryview(numpy.zeros(10, dtype=[("A", "m8[s]")]))

SystemError: <class 'memoryview'> returned a result with an error set

I have filed a bug with numpy, although I'm not quite sure that's where the problem lies.

Despite the ValueError, the assignment in `[20]` succeeded. Take a look at the variable `x` immediately after the assignment. You'll see that it has, in fact, been assigned a value. (Sorry, I don't know what is causing the ValueError.) — Warren Weckesser, Feb 24 '15 at 23:37
@WarrenWeckesser Huh, indeed it does. It didn't occur to me to check. — gerrit, Feb 24 '15 at 23:39
This looks like a bug in the `theading` module of python (try running the above code in a script without ipython), I've never seen anything like it before. — Bi Rico, Feb 25 '15 at 00:08
If I run the same sequence in a plain python shell, I don't get the ValueError in the assigment statement. The ValueError occurs when I try to show `x` by just typing the name and hitting return. So `>>> x = memoryview(...)` "works", but then `>>> x` raises the ValueError. — Warren Weckesser, Feb 25 '15 at 00:12
Where is the threading library involved? With the actual problem deep in my code (using `joblib`), the result is a traceback pointing at an entirely unrelated location, unless I step through using `ipdb`, which results in no related error at all. — gerrit, Feb 25 '15 at 00:38
There's something very strange going on here. The `ValueError` actually occurs when you execute *any* statement after the first one. — augurar, Aug 10 '16 at 07:40

augurar · Accepted Answer · 2016-08-10T08:08:27.410

There is something very odd going on here.

>>> memoryview(zeros(10, dtype=[("A", "m8[s]")]))
<memory at 0x102654348>
>>> 
ValueError: cannot include dtype 'm' in a buffer

My conjecture is that this is related to https://bugs.python.org/issue23571. Some C function underlying memoryview is both returning a non-null result and setting an error flag. This apparently causes the error to be raised when the next statement is executed! In Python 3.5, the interpreter raises a SystemError instead when this condition occurs.

It seems like the real bug here is with the memoryview function, not with numpy.

Why does creating this memoryview raise a ValueError only when assigning to a variable?

1 Answers1