0

16Gb machine gives out of memory error. I doubt if the conversion is really in place.

import numpy as np
x = np.ones(int(1.5e9), dtype=np.int64)  # 12 Gb
x.astype(np.float64, copy=False)  # gives out of memory error.

How to do in-place memory conversion? I want to convert data type and preserve the value. For example 1.0f becomes integer 1.

In-place type conversion of a NumPy array

hamster on wheels
  • 2,771
  • 17
  • 50

1 Answers1

4

About the copy parameter:

By default, astype always returns a newly allocated array. If this is set to false, and the dtype, order, and subok requirements are satisfied, the input array is returned instead of a copy.

So it's conditional.

In [540]: x=np.arange(10)
In [542]: x.dtype
Out[542]: dtype('int32')
In [543]: z=x.astype('float32',copy=False)
In [544]: z
Out[544]: array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.], dtype=float32)
In [545]: x.__array_interface__
Out[545]: 
{'data': (188221848, False),
 'descr': [('', '<i4')],
 'shape': (10,),
 'strides': None,
 'typestr': '<i4',
 'version': 3}
In [546]: z.__array_interface__
Out[546]: 
{'data': (191273640, False),
 'descr': [('', '<f4')],
 'shape': (10,),
 'strides': None,
 'typestr': '<f4',
 'version': 3}

z has different memory location.

The accepted answer in your link appears to work

In [549]: z=x.view('float32')
In [550]: z[:]=x
In [551]: z
Out[551]: array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.], dtype=float32)
In [552]: x
Out[552]: 
array([         0, 1065353216, 1073741824, 1077936128, 1082130432,
       1084227584, 1086324736, 1088421888, 1090519040, 1091567616])
In [553]: z
Out[553]: array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.], dtype=float32)
In [555]: x.__array_interface__
Out[555]: 
{'data': (188221848, False),
 'descr': [('', '<i4')],
 'shape': (10,),
 'strides': None,
 'typestr': '<i4',
 'version': 3}
In [556]: z.__array_interface__
Out[556]: 
{'data': (188221848, False),
 'descr': [('', '<f4')],
 'shape': (10,),
 'strides': None,
 'typestr': '<f4',
 'version': 3}

This works because z shares memory with x, but with a different dtype. When copied from x to z, they are converted to match the new dtype. Memory locations is preserved. However I can't guarantee that there wasn't a temporary buffer.

In case it isn't clear, conversion form int32 to float32 requires a change in the underlying bytes. The bit representation of integers is different from that of floats.

In [594]: np.array(1, 'int32').tobytes()
Out[594]: b'\x01\x00\x00\x00'
In [595]: np.array(1, 'float32').tobytes()
Out[595]: b'\x00\x00\x80?'
hpaulj
  • 221,503
  • 14
  • 230
  • 353
  • float32 and int32 both uses same amount of memory. but astype still makes new copy, which is not necessary and inefficient ... – hamster on wheels Jul 19 '17 at 22:35
  • What does efficiency have to do with it. Each 4byte block has to go through the same conversion process. Whether it gets written back to the same or different memory location shouldn't make much difference in 'speed'. – hpaulj Jul 19 '17 at 22:50
  • I guess making copy is slower than doing the operation in place because cpu don't have to read/write an extra cache line of memory. prefetch should mask memory latency. don't know if prefetch work better without that copying. https://stackoverflow.com/questions/3928995/how-do-cache-lines-work Making copy is less memory efficient for sure. – hamster on wheels Jul 19 '17 at 22:51
  • 1
    That memory cache might matter with very large arrays. I just did some timings. for `arange(10000)` the 2nd method is modestly faster; for `arange(1000000)` 15x. Times are still in the ns/µs range. – hpaulj Jul 19 '17 at 22:58
  • The `z[:]=x` rewrites all the data byes (inplace). `z` float is then the correct way to view them. `x.view('float32')` shows the proper `1.0`. – hpaulj Jul 19 '17 at 23:11
  • You are right. I missed that. At least numpy can work here. – hamster on wheels Jul 19 '17 at 23:13