Numpy reshape on view

Question

I'm confused about the results of numpy reshape operated on a view. In the following q.flags shows that it does not own the data, but q.base is neither x nor y, so what is it? I'm surprised to see that q.strides is 8 which means that it gets the next element by every time move 8 bytes in memory (if I understand correctly). However if none of the arrays other than x owns data, the only data buffer is from x, which does not permit getting the next element of q by moving 8 bytes.

In [99]: x = np.random.rand(4, 4)

In [100]: y = x.T

In [101]: q = y.reshape(16)

In [102]: q.base is y
Out[102]: False

In [103]: q.base is x
Out[103]: False

In [104]: y.flags
Out[104]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [105]: q.flags
Out[105]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [106]: q.strides
Out[106]: (8,)

In [107]: x
Out[107]: 
array([[ 0.62529694,  0.20813211,  0.73932923,  0.43183722],
       [ 0.09755023,  0.67082005,  0.78412615,  0.40307291],
       [ 0.2138691 ,  0.35191283,  0.57455781,  0.2449898 ],
       [ 0.36476299,  0.36590522,  0.24371933,  0.24837697]])

In [108]: q
Out[108]: 
array([ 0.62529694,  0.09755023,  0.2138691 ,  0.36476299,  0.20813211,
        0.67082005,  0.35191283,  0.36590522,  0.73932923,  0.78412615,
        0.57455781,  0.24371933,  0.43183722,  0.40307291,  0.2449898 ,
        0.24837697])

UPDATE:

It turns out that this question has been asked in the numpy discussion forum: http://numpy-discussion.10968.n7.nabble.com/OWNDATA-flag-and-reshape-views-vs-copies-td10363.html

Answer 1

In short: you cannot always rely on the ndarray.flags['OWNDATA'].

>>> import numpy as np
>>> x = np.random.rand(2,2)
>>> y = x.T
>>> q = y.reshape(4)
>>> y[0,0]
0.86751629121019136
>>> y[0,0] = 1
>>> q
array([ 0.86751629,  0.87671107,  0.65239976,  0.41761267])
>>> x
array([[ 1.        ,  0.65239976],
       [ 0.87671107,  0.41761267]])
>>> y
array([[ 1.        ,  0.87671107],
       [ 0.65239976,  0.41761267]])
>>> y.flags['OWNDATA']
False
>>> x.flags['OWNDATA']
True
>>> q.flags['OWNDATA']
False
>>> np.may_share_memory(x,y)
True
>>> np.may_share_memory(x,q)
False

Because q didn't reflect the change in the first element, like x or y, it must somehow be the owner of the data (somehow is explained below).

There is more discussion about the OWNDATA flag over at the numpy-discussion mailinglist. In the How can I tell if NumPy creates a view or a copy? question, it is briefly mentioned that simply checking the flags.owndata of an ndarray sometimes seems to fail and that it seems unreliable, as you mention. That's because every ndarray also has a base attribute:

the base of an ndarray is a reference to another array if the memory originated elsewhere (otherwise, the base is None). The operation y.reshape(4) creates a copy, not a view, because the strides of y are (8,16). To get it reshaped (C-contiguous) to (4,), the memory pointer would have to jump 0->16->8->24, which is not doable with a single stride. Thus q.base points to the memory location generated by the forced-copy-operation y.reshape, which has the same shape as y, but copied elements and thus has normal strides again: (16, 8). q.base is thus not bound to by any other name as it was the result of the forced-copy operation y.reshape(4). Only now can the object q.base be viewed in a (4,) shape, because the strides allow this. q is then indeed a view on q.base.

For most people it would be confusing to see that q.flags.owndata is False, because, as shown above, it is not a view on y. However, it is a view on a copy of y. That copy, q.base, is the owner of the data however. Thus the flags are actually correct, if you inspect closely.

Answer 2

I like to use .__array_interface__.

In [811]: x.__array_interface__
Out[811]: 
{'data': (149194496, False),
 'descr': [('', '<f8')],
 'shape': (4, 4),
 'strides': None,
 'typestr': '<f8',
 'version': 3}

In [813]: y.__array_interface__
Out[813]: 
{'data': (149194496, False),
 'descr': [('', '<f8')],
 'shape': (4, 4),
 'strides': (8, 32),
 'typestr': '<f8',
 'version': 3}

In [814]: x.strides
Out[814]: (32, 8)
In [815]: y.strides
Out[815]: (8, 32)

Transpose was performed by reversing the strides. The base data pointer is the same.

In [817]: q.__array_interface__
Out[817]: 
{'data': (165219304, False),
 'descr': [('', '<f8')],
 'shape': (16,),
 'strides': None,
 'typestr': '<f8',
 'version': 3}

So the q data is a copy (different pointer). Strides (8,) means its elements are accessed by stepping from one f8 to the next. But a x.reshape(16) is a view of x - because its data can be accessed with a simple 8 step.

To access the original data in the q order, it would have to step 32 bytes 3 times (down x rows), then go back to the start and step 8 to the 2nd x column, followed by 3 row steps, etc. Since striding doesn't work this way, it has to work from a copy.

Note also that y[0,0] changes x[0,0], but q[0] is independent of both.

While OWNDATA for q is false, it is True for y.ravel() and y.flatten(). I suspect reshape() in this case is making a copy, and then reshaping, and it's the intermediate copy that 'owns' the data, q.base.