0

I'm am trying to extract real values coming from a fortran file using np.loadtxt, the name of the file is fort.31 It seems a number is not taken into account by python: 0.22738+109

Do you have any idea on how to get over it?

Here is my code

vraisemblance2d=np.loadtxt("fort.31")

And here is the complete error i get

Traceback (most recent call last): 
 File "plots3d.py", line 28, in <module> 
   vraisemblance2d=np.loadtxt("fort.31") 
 File "/home/myname/anaconda3/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1146, in loadtxt 
   for x in read_data(_loadtxt_chunksize): 
 File "/home/myname/anaconda3/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1074, in read_data 
   items = [conv(val) for (conv, val) in zip(converters, vals)] 
 File "/home/myname/anaconda3/lib/python3.7/site-packages/numpy/lib/npyio.py", line 1074, in <listcomp> 
   items = [conv(val) for (conv, val) in zip(converters, vals)] 
 File "/home/myname/anaconda3/lib/python3.7/site-packages/numpy/lib/npyio.py", line 781, in floatconv 
   return float(x) 
ValueError: could not convert string to float: '0.22738+109'
mulaixi
  • 183
  • 2
  • 11
  • 1
    This is a huge number (almost the number of atoms in the universe). Can you scale the numbers without effecting your result or do you have to use this number? for example using log*? – Tamir Nov 17 '20 at 08:47
  • I have tried, and i obtain some Nan (Not a number) in fort.31 Which are not really usable –  Nov 17 '20 at 08:59
  • 1
    The magnitude of the number isn't the problem; `float` can produce double floats, which have a maximum magnitude of about 1e308. The problem is that the letter `e` is missing; it should be `0.22738e+109`. How was the file created? Are all of the numbers in the file missing the `e`? If you cannot regenerate the file, you might need to resort to string operations in Python, i.e. parse the parts of the number and glue it back together with the `e`, and then apply `float`. – Robert Dodier Nov 17 '20 at 08:59
  • Do you know the `shape` of the input array before you load it? Or at least how many columns? – Daniel F Nov 17 '20 at 09:05
  • The file was created using the following fortran line WRITE(31,'(G12.5,G12.5,G12.5,2XG12.5)') x,y,z,max(like,1.e-20) where x,y,z describe a regular mesh and the likelihood (like) The likelihood takes some values without an e, e.g. 0.17556+139 or 0.14270+309 or even Infinity... How is it possible to have those numbers encoded right? –  Nov 17 '20 at 09:09
  • 1
    OK, assume I know nothing about `fortran` - do you know how many columns you have? Sounds like 3, or maybe 5? – Daniel F Nov 17 '20 at 09:11
  • @DanielF There are 4 columns and 64 481 201 lines... –  Nov 17 '20 at 09:12

1 Answers1

0

OK, four columns. I know nothing about fortran, so I'm just going to handle it on the python side. I'm assuming utf encoding but that might need to be tuned:

conv =  lambda x: float("e+".join(x.split("+")))   
vraisemblance2d = np.loadtxt("fort.31", 
                             converters = {i : conv for i in range(4)},
                             encoding   = "utf-8")

If that doesn't work, and fortran is giving you a mix of #.###+### and #.###e+### format, you'll need regex.

import re

reg = re.compile('(\d\.\d+)\+(\d+)')

def conv(x):
    try:
        return float(x)
    except:
        return float("e+".join(reg.match(x).groups())) 

vraisemblance2d = np.loadtxt("fort.31", 
                             converters = {i : conv for i in range(4)},
                             encoding   = "utf-8")

WARNING: I am no regex wizard, there's probably a better & faster way to do that.

Daniel F
  • 13,620
  • 2
  • 29
  • 55
  • Thanks for your answer, i get two errors Here is the first error Traceback (most recent call last):  Huge blablabla  File "plots3d.py", line 28, in    conv = lambda x: float("e+".join(x.split("+"))) TypeError: a bytes-like object is required, not 'str' –  Nov 17 '20 at 09:47
  • Here is the second File "plots3d.py", line 23, in conv    return float(("e+".join(reg.match(x).groups()))) TypeError: cannot use a string pattern on a bytes-like object –  Nov 17 '20 at 09:48
  • Oh man, what `python` version are you using? – Daniel F Nov 17 '20 at 09:48
  • Here is the answer of sys.version 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0] –  Nov 17 '20 at 09:51
  • Oh wait, it's `loadtxt` outputting `bytes` for compatibility to old `python`. Check edit. – Daniel F Nov 17 '20 at 10:00
  • Yes, i do not know how to apply the function to the file. loadtxt allowed me to have the output –  Nov 17 '20 at 10:05
  • That last dupe target has a recipe for `np.genfromtext`, I'd say use that – Daniel F Nov 17 '20 at 10:09
  • np.genfromtxt("fort.31") is Killed.. –  Nov 17 '20 at 10:17