Why do I get an overflow error multiplying Numpy product outputs?

Question

I've encountered an overflow warning as a result of multiplying the output of Numpy products that I'm looking to understand. A simplified version of their actual use within my larger project is detailed below:

import numpy as np

class MyClass(object):
    def __init__(self,
                 array_1,
                 array_2):

        # Assigning arrays to be used in later methods
        self.array_1 = array_1
        self.array_2 = array_2

        # Assigning some scaling factors to be used in later methods.
        self.value_1 = np.prod(self.array_1.shape)
        self.value_2 = np.prod(self.array_2.shape)
        print("Numpy Product Assignment: {0}, {1}".format(self.value_1, self.value_2))

        # Alternative assignment of scaling factors
        self.alt_value_1 = self.array_1.shape[0] * self.array_1.shape[1]
        self.alt_value_2 = self.array_2.shape[0] * self.array_2.shape[1]
        print("Regular Product Assignment: {0}, {1}".format(self.alt_value_1, self.alt_value_2))

        pass

    def mymethod(self):
        print("Direct Multiplication: {0}".format(80160 * 262144))
        print("Numpy Product Multiplication: {0}".format(self.value_1 * self.value_2))
        print("Regular Product Multiplcation {0}".format(self.alt_value_1 * self.alt_value_2))

if __name__ == '__main__':
    test_array_1 = np.zeros([512, 512], dtype=complex)
    test_array_2 = np.zeros([1002, 80], dtype=complex)

    test_class = MyClass(test_array_1, test_array_2)
    test_class.mymethod()

Including the use with the class structure for completeness, though heavily edited down to the bare minimum. If I run this code (on Python 3.6.0) I get the following output:

C:/somepath/scratch.py:247: RuntimeWarning: overflow encountered in long_scalars
  print("Numpy Product Multiplication: {0}".format(self.value_1 * self.value_2))
Numpy Product Assignment: 262144, 80160
Regular Product Assignment: 262144, 80160
Direct Multiplication: 21013463040
Numpy Product Multiplication: -461373440
Regular Product Multiplcation 21013463040

Process finished with exit code 0

Clearly I can get around the problem using the regular multiplication, but I would like to understand why there is a problem and if it can be fixed as is. I think there is some dtype=X subtlety that I have missed, so my question is what is causing these overflow errors?

I think the problem occurs because `numpy` uses c-type integers internally, while the regular multiplication uses `python` integers, which have a much larger [maximum value](https://stackoverflow.com/a/7604981/2454357). — Thomas Kühn, Jan 09 '18 at 11:57
See also [this answer](https://stackoverflow.com/a/23189557/2454357). — Thomas Kühn, Jan 09 '18 at 11:59
@ThomasKühn "Much larger maximum value" Indeed. Integers have in Python3 theoretically no maximum value, only your system sets the limits. — Mr. T, Jan 09 '18 at 12:35
@Piinthesky Yes, I formulated it badly, but the answer I linked includes that information. — Thomas Kühn, Jan 09 '18 at 12:39

score 5 · Accepted Answer · edited Jul 11 '20 at 18:33

5

This looks like an overflow caused by 32 integers. You can convert your values to 64 bit like this:

    self.value_1 = np.prod(self.array_1.shape, dtype=np.int64)
    self.value_2 = np.prod(self.array_2.shape, dtype=np.int64)

Numpy automatically selects a 32 bit integer type if the values used for array construction are small enough. During multiplication they are not automatically cast to 64 bit.

edited Jul 11 '20 at 18:33

Anand

33
1
7

answered Jan 09 '18 at 11:58

MB-F

22,770
4
61
116

So it uses the size of the np.prod result to determine the output dtype, not any properties of the input array, unless you have defined the dtype argument within np.prod? – Folau Jan 09 '18 at 12:03
1

Actually, `prod`normally uses the properties of the input array - but you pass in `array.shape`, which is a tuple. The tuple needs to be converted to an array, and this conversion looks at the values. (Oh, and it looks like `prod` takes a `dtype` argument that you can use instead of calling `.astype` on the result.) – MB-F Jan 09 '18 at 12:23
Ah I had missed that shape is a tuple, but that does make sense. Thanks. – Folau Jan 09 '18 at 13:10

Why do I get an overflow error multiplying Numpy product outputs?

1 Answers1