2

I have to do some math operations (e.g., add, multiply) on a large array. To prevent any 'MemoryError' , I am doing my computations as suggested on the answer from this thread.

However, I am running into some trouble while applying the assignment operations as per suggested in the thread. I will demonstrate my problem using a small 3x3 array.

I have the following input array K:

array([[ 0.        ,  0.51290339,  0.24675368],
       [ 0.51290339,  0.        ,  0.29440921],
       [ 0.24675368,  0.29440921,  0.        ]])

I want to apply the following computation to the input array K:

output = K* (1.5 - 0.5 * K* K)

I apply the above equation to compute the desired output as follows in Python:

K*= (1.5+np.dot(np.dot(-0.5,K),K))

However, the output answer is not correct.

My desired answer should be:

0.0000000 0.7018904 0.3626184
0.7018904 0.0000000 0.4288546
0.3626184 0.4288546 0.0000000

Any help is welcome.

user121
  • 849
  • 3
  • 21
  • 39
  • Related https://stackoverflow.com/questions/40034993/how-to-get-element-wise-matrix-multiplication-hadamard-product-in-numpy, https://stackoverflow.com/questions/29254123/is-dot-product-and-normal-multiplication-results-of-2-numpy-arrays-same – Ilja Everilä Jan 05 '18 at 16:22

3 Answers3

1

The difference arises because dot computes the dot product whereas * computes the element-wise product. Try using

K *= 1.5 - 0.5 * K * K

instead.

Addition

Unfortunately, that does not yet solve the memory problems. I would recommend using cython to compute the desired function without allocating extra memory.

# This cython function must be compiled
def evaluate_function_inplace(double[:] values):
    cdef int i
    for i in range(values.shape[0]):
        values[i] *= 1.5 - 0.5 * values[i] * values[i]

Subsequently, you can use the function like so.

K = ...
evaluate_function_inplace(K.ravel())

The K.ravel() call will flatten the array but will not allocate new memory.

Of course, you can also use the above approach without resorting to cython but the performance overhead of iterating over the elements of such a large array in python are very large.

Till Hoffmann
  • 9,479
  • 6
  • 46
  • 64
  • Till, this solution works fine for relatively small arrays....... My array is (29000 x 29000). When I use the above function it gives MemoryError. Any advice? – user121 Jan 05 '18 at 14:28
  • Unfortunately, at some point you have to compute the square of `K` (and store it in memory) to be able to multiply. You can work around the need to store the entire array if you compute the values in place (see https://stackoverflow.com/a/6824389/1150961 for a discussion). I've added a possible solution using cython. – Till Hoffmann Jan 05 '18 at 14:39
  • I am looking for a non-Cython solution. – user121 Jan 05 '18 at 15:52
0

Your problem is that you're actually performing matrices multiplications. In your case what you want is the following : 

K = (np.dot(-0.5,K) * K + 1.5) * K
Merkle Daamgard
  • 86
  • 8
0

Try this

 K*= (1.5+np.multiply(np.multiply(-0.5,K),K))

It gives output

array([[ 0.        ,  0.70189037,  0.36261843],
       [ 0.70189037,  0.        ,  0.42885459],
       [ 0.36261843,  0.42885459,  0.        ]])
Artier
  • 1,648
  • 2
  • 8
  • 22