Aggregation Of Pandas Groupby Objects

I am trying to aggregate some statistics from a groupby object on chunks of data. I have to chunk the data because there are many (18 million) rows. I want to find the number of ro

Solution 1:

Call add and pass fill_value=0 you could iteratively add whilst chunking I guess:

In [98]:

df = pd.DataFrame({'X': ['A','B','C','A','B','C','B','C','D','B','C','D'],
                       'Y': np.arange(12)})
df[0:6].groupby(['X']).count().add(df[6:].groupby(['X']).count(), fill_value=0)
Out[98]:
   Y
X   
A  2
B  4
C  4
D  2