Reading multiple CSV files with headers in python with numpy?

Question

Currently I am using this code to read one complete csv file to my code:

data = np.loadtxt('csv_Complete.csv', delimiter=',', skiprows=1)

However, Now I have multiple csv files that are in this format.

Log1.csv

x1,x2,x3,x4....
1.5,3,5,7,8
2,5,1.2,5,2
1,3,3,5.5,6

log2.csv

 x1,x2,x3,x4....
 1,3.3,5,7,8
 2,5.1,1,5.5,2
 1,3,3,5,6

This is the method I am thinking of doing but it is not working. Getting a ValueError: could not convert string to float:

log1 = np.loadtxt('log1.csv', delimiter=',', skiprows=1)
log2 = np.loadtxt('log2.csv', delimiter=',', skiprows=1)
log3 = np.loadtxt('log3.csv', delimiter=',', skiprows=1)
data = np.append([log1, log2, log3])

The error I am getting is:

 File "<ipython-input-6-6155c8de61ad>", line 1, in <module>
    runfile('C:/Users/Mmyname/.spyder2-py3/setdataexp.py', wdir='C:/Users/myname/.spyder2-py3')

  File "C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 89, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/myname/.spyder2-py3/setdataexp.py", line 5, in <module>
    log1 = np.loadtxt('log40a.csv', delimiter=',', skiprows=1)

  File "C:\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 930, in loadtxt
    items = [conv(val) for (conv, val) in zip(converters, vals)]

  File "C:\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 930, in <listcomp>
    items = [conv(val) for (conv, val) in zip(converters, vals)]

  File "C:\Anaconda3\lib\site-packages\numpy\lib\npyio.py", line 659, in floatconv
    return float(x)

ValueError: could not convert string to float:

Both of your example files load fine for me. Are you sure all three input files have only a single line of header, and all the rest is a proper csv until the end? I doubt it. Also, check where the error is coming from: it should be one of the loadtxt calls, telling you which file is problematic. — Andras Deak -- Слава Україні
– Andras Deak -- Слава Україні, Commented Oct 10, 2016 at 16:44
It might be an issue with a blank line at the end of your file... just a guess. — kpie
– kpie, Commented Oct 10, 2016 at 17:08
Are you sure you didn't miss a few final lines of the traceback...? Anyway, look at log40a.csv. That's the problematic one. — Andras Deak -- Слава Україні
– Andras Deak -- Слава Україні, Commented Oct 10, 2016 at 17:09
@AndrasDeak No that is all in the traceback. I just tried switching it so it reads the log40b first and I get the same error with log40b being the problem. — J. Jones
– J. Jones, Commented Oct 10, 2016 at 17:15

torm · Accepted Answer · 2016-10-10 17:28:04Z

1

It must be a missing value in file log40a.csv.

I have the same error for file like:

x1,x2,x3,x4....
1,3.3,5,7,8
2,5.1,,5.5,2
1,3,3,5,6

Base on documentation if you have missing values you should use genfromtxt function.

edited Oct 10, 2016 at 17:28

answered Oct 10, 2016 at 17:19

torm

1,54615 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

J. Jones Over a year ago

Yes I do have some missing values. Is it possible to have it skip it?

J. Jones Over a year ago

Actually, I would rather it uses the last value in the same row as the missing value is that possible?

torm Over a year ago

You can find all missing values, replace them with last in row and than use loadtxt function. You need to prepare your data files at first.

J. Jones Over a year ago

@AndrasDeak I looked at the data again. There is no missing values. There is a gap in time but that doesn't translate to missing values in the columns or rows. So rows and columns are filled with something. That brings me back to square 1.

torm Over a year ago

@J.Jones You call append function in wrong way. Here you find more information.

|

Collectives™ on Stack Overflow

Reading multiple CSV files with headers in python with numpy?

1 Answer 1

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related