Python numpy nan compare to true in arrays with strings

Question

I am trying to compare two numpy arrays which contain numbers, string and nans. I want to know how many items in the array are equal.

When comparing these two arrays:

c =np.array([1,np.nan]);
d =np.array([2,np.nan]);
print (c==d)
[False False]

Which is the expected behaviour.

However, when comparing:

a =np.array([1,'x', np.nan]);
b =np.array([1,'x', np.nan]);
print (a==b)
[ True  True  True]

That makes no sense to me, how can adding a string to the array change the way nans are compared? Any ideas?

Thanks!

NPE · Accepted Answer · 2014-11-29 17:59:44Z

1

If you examine the arrays, you'll see that np.nan has been converted to string ('n'):

In [48]: a = np.array([1, 'x', np.nan])

In [49]: a
Out[49]: 
array(['1', 'x', 'n'], 
      dtype='|S1')

And 'n' == 'n' is True.

What I don't understand is why changing the array's dtype to object doesn't change the result of the comparison:

In [72]: a = np.array([1, 'x', np.nan], dtype=object)

In [73]: b = np.array([1, 'x', np.nan], dtype=object)

In [74]: a == b
Out[74]: array([ True,  True,  True], dtype=bool)

In [75]: a[2] == b[2]
Out[75]: False

In [76]: type(a[2])
Out[76]: float

In [77]: type(b[2])
Out[77]: float

It's almost as if the two NaN objects are compared by reference rather than by value:

In [79]: id(a[2])
Out[79]: 26438340

In [80]: id(b[2])
Out[80]: 26438340

edited Nov 29, 2014 at 17:59

answered Nov 29, 2014 at 17:53

NPE

503k114 gold badges970 silver badges1k bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Daniel Over a year ago

In numpy 1.9.1 when comparing your a and b I get "FutureWarning: numpy equal will not check object identity in the future. The comparison did not return the same result as suggested by the identity (is)) and will change.". Looks like this is a known issue (feature?).

sebix Over a year ago

Well, comparing NaN as an object, I except it to be equal. NaN as a float should be always False. But don't now the Numpy-internal very well.

NPE Over a year ago

@sebix: I am afraid I don't understand the distinction you're making, given that float is a subclass of object (try isinstance(float, object)).

sebix Over a year ago

I assume that with object-arrays, Numpy won't use the objects methods to compare, but the ids like you assume.

NPE Over a year ago

@sebix: I can demonstrate that this is not the case: an array with dtype of object containing strings uses string contents -- and not just ids -- to establish equality.

|

Collectives™ on Stack Overflow

Python numpy nan compare to true in arrays with strings

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related