现在的位置: 首页 > 问答 > 正文

Pandas: removing rows with NaN in MultiIndex (with duplicates)

2017年09月14日 问答 ⁄ 共 1165字 ⁄ 字号 暂无评论

Updated with a DataFrame that repros my exact issue

I have an issue where NaN appearing in my indexes is leading to non-unique rows (since NaN !== NaN). I need to drop all rows where NaN occurs in the index. My previous question had an example DataFrame with a single NaN row, however the original solution did not resolve my issue as it did not meet this poorly advertised requirement:

(Note that in the actual data I have thousands of such rows, including duplicate rows since NaN !== NaN so this is permissible on an index)

(from my original post)

The Issue

>>>import pandas as pd
>>>import numpy as np
>>> df = pd.DataFrame([[1,1,"a"],[1,2,"b"],[1,3,"c"],[1,np.nan,"x"],[1,np.nan,"x"],[1,np.nan,"x"],[2,1,"d"],[2,2,"e"],[np.nan,1,"x"],[np.nan,2,"x"],[np.nan,1,"x"]], columns=["a","b","c"])
>>>df
         c
a   b
1.0 1.0  a
    2.0  b
    3.0  c
    NaN  x
    NaN  x
    NaN  x
2.0 1.0  d
    2.0  e
NaN 1.0  x
    2.0  x
    1.0  x

Note the duplicate rows: (1.0, NaN) and (NaN, 1.0)

Failed Solutions:

I've tried something simple like:

>>>df = df[pandas.notnull(df.index)]

But this fails because notnull is not implemented for MultiIndex.

Also one of the early answers suggested:

>>>df = df.reindex(df.index.dropna())

However this failed with the error:

Exception: cannot handle a non-unique multi-index!

Desired Output:

>>>df
         c
a   b
1.0 1.0  a
    2.0  b
    3.0  c
2.0 1.0  d
    2.0  e

(all NaN index rows are dropped, eliminating any non-unique rows)

给我留言

您必须 [ 登录 ] 才能发表留言!

×