numpy array dimension mismatch error

Question

I am quite new to numpy and python in general. I am getting a dimension mismatch error when I try to append values even though I have made sure that both arrays have the same dimension. Also another question I have is why does numpy create a single dimensional array when reading in data from a tab delimited text file.

import numpy as np

names = ["Angle", "RX_Power", "Frequency"]
data = np.array([0,0,0],float)  #experimental

data = np.genfromtxt("rx_power_mode 0.txt", dtype=float, delimiter='\t', names = names, usecols=[0,1,2], skip_header=1)

freq_177 = np.zeros(shape=(data.shape))

print(freq_177.shape)  #outputs(315,)

for i in range(len(data)):

    if data[i][2] == 177:
        #np.concatenate(freq_177,data[i]) has same issue
        np.append(freq_177,data[i],0)

The output I am getting is

all the input arrays must have same number of dimensions

Can you output the shape of your data? If your data is the same shape as freq_177 you shouldn't be able to index a second dimension — jfish003
– jfish003, Commented Dec 18, 2015 at 0:23
the data shape is (315,) data[i] is (0.0, -86.0, 177.0) at i = 0 — DBB
– DBB, Commented Dec 18, 2015 at 0:43
I think this happens due to getfromtxt as I try to declare data as multidimentional — DBB
– DBB, Commented Dec 18, 2015 at 0:46

Community · Accepted Answer · 2020-06-20 09:12:55Z

1

Annotated code:

import numpy as np

names = ["Angle", "RX_Power", "Frequency"]

You don't need to 'initialize' an array - unless you are going to assign values to individual elements.

data = np.array([0,0,0],float)  #experimental

This data assignment completely overwrites the previous one.

data = np.genfromtxt("rx_power_mode 0.txt", dtype=float, delimiter='\t', names = names, usecols=[0,1,2], skip_header=1)

Look at data at this point. What is data.shape? What is data.dtype? Print it, or at least some elements. With names I'm guessing that this is a 1d array, with a 3 field dtype. It's not a 2d array, though, with all floats it could transformed/view as such.

Why are you making a 1d array of zeros?

freq_177 = np.zeros(shape=(data.shape))
print(freq_177.shape)  #outputs(315,)

With a structured array like data, the preferred way to index a given element is by field name and row number, eg. data['frequency'][i]`. Play with that.

np.append is not the same as the list append. It returns a value; it does not change freq_177 in place. Same for concatenate. I recommend staying away from np.append. It's too easy to use it in the wrong way and place.

for i in range(len(data)):
    if data[i][2] == 177:
        #np.concatenate(freq_177,data[i]) has same issue
        np.append(freq_177,data[i],0)

It looks like you want to collect in freq_177 all the terms of the data array for which the 'frequency' field is 177.

I = data['frequency'].astype(int)==177
freq_177 = data[I]

I have used astype(int) because the == test with floats is uncertain. It is best used with integers.

I is a boolean mask, true where the values match; data[I] then is the corresponding elements of data. The dtype will match that of data, that is, it will have 3 fields. You can't append or concatenate it to an array of float zeros (your original freq_177).

If you must iterate and collect values, I suggest using list append, e.g.

alist = []
for row in data:
   if int(row['frequency'])==177:
       alist.append(row)
freq177 = np.array(alist)

I don't think np.append is discussed much except in its own doc page and text. It comes up periodically in SO questions.

http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.append.html

Returns: append : ndarray

A copy of arr with values appended to axis. Note that append does not occur in-place: a new array is allocated and filled.

See also help(np.append) in an interpreter shell.

For genfromtxt - it too has docs, and lots of SO discussion. But to understand what it returned in this case, you need to also read about structured arrays and compound dtype. (add links?)

Try loading the data with:

data = np.genfromtxt("rx_power_mode 0.txt", dtype=float, delimiter='\t', usecols=[0,1,2], skip_header=1)

Since you are skipping the header line, and just using columns with floats, data should be a 2d array with 3 columns, (N, 3). In that case you could access the 'frequency' values with data[:,2]

I = int(data[:,2])==177
freq_177 = data[I,:]

freq_177 is now be a 3 column array - with a subset of the data rows.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Dec 18, 2015 at 1:28

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

DBB Over a year ago

Hi Thanks for the reply. That worked really well. I had a few more questions. First i m not sure why does genfromtxt return a 1d array. Second could you point out to a good reference for numpy. I was looking at Scipy.org for function append but all it says is that it appends values to the end of an array. Could have never figured out this solution on my own.

hpaulj Over a year ago

I added some notes on documentation. I should also have suggested how you could load your data so it produces a 2d array (assuming it's all numbers).

DBB Over a year ago

Ahhh now I understand how you created a boolean array and filtered out the results. So after going through the documentation for genfromtxt I see that it returns an nd array. But the weird thing is even loadtxt returns an ndarray. But if I use loadtxt on my example file(as I have no missing values for now) it does end up returning a 2d array. Maybe it has to do with the way it processes data. In any case your additional comments were super helpful. Thanks a lot.

Collectives™ on Stack Overflow

numpy array dimension mismatch error

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related