Cant convert to different data types

Question

I am trying to run a .txt file with raw data into my code but I keep getting a value error. It was working before, but now I am getting this error:

ValueError: could not convert string to float: '.'

Here is my file with the raw data:

0.0980224609375
0.10589599609375
0.0980224609375
0.0980224609375
0.0980224609375
0.11767578125
0.130.0980224609375    --> The error is here I assume since there are 2 periods
0.10198974609375
0.10198974609375
0.0980224609375

This data can not be changed, so how can I convert this from a string to float without getting an error? Here is my code:

# Read and pre-process input images
n, c, h, w = net.inputs[input_blob].shape
images = np.ndarray(shape=(n, c, h, w))
for i in range(n):
    image = cv2.imread(args.input[i])
    if image.shape[:-1] != (h, w):
        log.warning("Image {} is resized from {} to {}".format(args.input[i], image.shape[:-1], (h, w)))
        image = cv2.resize(image, (w, h))
    # Swapping Red and Blue channels 
    #image[:, :, [0, 2]] = image[:, :, [2, 0]]
    # Change data layout from HWC to CHW
    image = image.transpose((2, 0, 1))  
    images[i] = image
    
    eoim = image
    eoim16 = eoim.astype(np.float16)
    
    # divide by 255 to get value in range 0->1 if necessary (depends on input pixel format)
    if(eoim16.max()>1.0):
        eoim16 = np.divide(eoim16,255)
        print(eoim16)

val = []
preprocessed_image_path = 'C:/Users/Owner/Desktop/Ubotica/IOD/cloud_detect/'
formated_image_file = "output_patch_fp"
f = open(preprocessed_image_path + "/" + formated_image_file + ".txt", 'r')
'''elem_counter = 0
for elem in eoim16:
    for elem1 in elem:
        for col in elem1:
            #f.read(int(float(formated_image_file)))
            val = float(f.readline())'''
for y in f.readlines()[0]:
    val.append(float(y))
f.close()

#print(val)
#val = np.reshape(val, (3,512,512))
val = np.ndarray(shape=(c, h, w))
#res = val

# calling the instance method using the object cloudDetector
res = cloudDetector.infer(val)
res = res[out_blob]

Any help will be much appreciated!

What does it even mean to convert `0.130.0980224609375` to a float? Neither I nor Python has any idea. It seems like whatever program generated that file has a bug, so the bug you need to fix is upstream from your code. The answer to "how can I convert this from a string to float" is -- you can't. It is throwing an error because this is a genuine error situation. — John Coleman, Jul 23 '20 at 16:16
How would `0.130.0980224609375` be converted? What number does it represent? — wwii, Jul 23 '20 at 16:17
What float value would you expect `0.130.0980224609375` to be converted to? How do you know that none of the other values have been similarly corrupted, just not in a way that produced a syntactially invalid number? — jasonharper, Jul 23 '20 at 16:17
Im just taking raw data from this file and reading it into my code. When I do I get ValueError: could not convert string to float: '.' — Tristan, Jul 23 '20 at 16:18
0.130.0980224609375 is a float value that is being passed and read into my code but I get a value error because it has 2 dots. If i remove it, it works, but this is data that is given to me so I can not change the data — Tristan, Jul 23 '20 at 16:20
But -- that isn't a float value. That is just some digits with a couple of periods. If the rest of the data is good, this should be made an `NA` value rather than a float. Libraries like `pandas` are designed to handle `NA`s — John Coleman, Jul 23 '20 at 16:22
What number does it represent? Can you skip it - not process it? — wwii, Jul 23 '20 at 16:22
its suppose to represent a float value, I have no idea why double dots appear, I am just trying to get rid of one dot so it can be represented as a float value — Tristan, Jul 23 '20 at 16:29
I see you are a new contributor. If that means you are new to programming and/or Python it is likely that the problems you run into are not unique, have been encountered before by other programmers and probably been asked about and answered here on SO. Thinking about your problem a bit and trying a few different search terms often is productive. — wwii, Jul 23 '20 at 16:37
@Tristan the file is clearly generated by a computer rather than typed by hand, so the double periods are not typos. Perhaps they mean something. Did you read the specification of the files? Maybe it discusses it. If the computer that generates the file is really trying to write floats but inserts stray periods, do you really have any reason to trust any of the numbers? Writing float literals to a text file is easy. It shouldn't have such a bug. Note that if you simply remove the second dot -- the resulting string has too many digits to be a float literal like the other ones. — John Coleman, Jul 23 '20 at 16:38
A final observation: if that is supposed to be `0.1300980224609375 ` then it would be an outlier relative to the other numbers you've shown (which cluster around `0.10`). Are the other double-period strings similarly unusual numbers if you interpret them as floats by dropping the second period? — John Coleman, Jul 23 '20 at 16:49

score 0 · Answer 1 · answered Jul 23 '20 at 16:20

0

You've correctly identified what's going wrong. 0.130.0980224609375 confuses Python, and would confuse most humans as well. Does it mean 0.13009...? Does it mean 0.130? Is it 2 decimal numbers? Is it an ip address? Python doesn't do much thinking, just shrugs its shoulders and quits. This code will assume you mean one decimal.

def clean(s):
    while s.count(".") > 1:
        i = s.rindex(".")
        s = s[:i] + s[i+1:]
    return s

assert clean("0.130.0980224609375") == "0.1300980224609375"

answered Jul 23 '20 at 16:20

JimmyCarlos

1,934
1
10
24

For this data, it does mean one decimal place. There are other instances of where this double-dot appears (I just showed a small piece of data where the problem occurred), so if there are other instances of this dot, do I have to assert clean each one? Each time I run the program new data appears within the .txt file with similar double dot issues, so I do not know if assert clean would work since you have to specify the number?? unless I am mistaken – Tristan Jul 23 '20 at 16:25

ACarter · Answer 2 · 2020-07-23T16:34:07.623

0

So the error is because of that extra dot in the number that means it is not a valid float.

If that extra dot was there by mistake, and you wanted to remove it, you could use:

index = y.find('.', 2) // find if there is a '.' in the string, at a position greater than 2

if(index != -1): // if there is a '.' found (find() returns -1 if the character is not found)

    y = y[:index] + y[index+1:] // reconstruct the string from the section before the . and the section after

val.append(float(y))

Alternatively, if you just wanted to ignore it:

try:
    val.append(float(y))
except ValueError:
    pass

edited Jul 23 '20 at 16:34

answered Jul 23 '20 at 16:21

ACarter

5,688
9
39
56

How do you know that the "extra dot" is safe to remove? It is arbitrary to get a float literal out of a non-float literal the way that you propose. – John Coleman Jul 23 '20 at 16:24
you are totally right. I assumed that it was there in error, which you are right, may not be a valid assumption – ACarter Jul 23 '20 at 16:33

Cant convert to different data types

2 Answers2