How To Speed Up Loop In Numpy?

I would like to speed up this code : import numpy as np import pandas as pd a = pd.read_csv(path) closep = a['Clsprc'] delta = np.array(closep.diff()) upgain = np.where(delta &g

Solution 1:

It looks like you're trying to calculate an exponential moving average (rolling mean), but forgot the division. If that's the case then you may want to see this SO question. Meanwhile, here's a fast a simple moving average using the cumsum() function taken from the referenced link.

defmoving_average(a, n=14) :
    ret = np.cumsum(a, dtype=float)
    ret[n:] = ret[n:] - ret[:-n]
    return ret[n - 1:] / n

If this is not the case, and you really want the function described, you can increase the iteration speed by getting using the external_loop flag in your iteration. From the numpy documentation:

The nditer will try to provide chunks that are as large as possible to the inner loop. By forcing ‘C’ and ‘F’ order, we get different external loop sizes. This mode is enabled by specifying an iterator flag.

Observe that with the default of keeping native memory order, the iterator is able to provide a single one-dimensional chunk, whereas when forcing Fortran order, it has to provide three chunks of two elements each.

forxinnp.nditer(upgain[14:],  flags=['external_loop'], order='F'):
    # xnowhasx[0],x[1], x[2], x[3], x[4], x[5]elements.

Solution 2:

In simplified terms, I think this is what the loops are doing:

upgain=np.array([.1,.2,.3,.4])    
u=[]
up=1
for x in upgain:                  
    u1=10*up+x
    u.append(u1)
    up=u1

producing:

[10.1, 101.2, 1012.3, 10123.4]

np.cumprod([10,10,10,10]) is there, plus a modified cumsum for the [.1,.2,.3,.4] terms. But I can't off hand think of a way of combining these with compiled numpy functions. We could write a custom ufunc, and use its accumulate. Or we could write it in cython (or other c interface).

https://stackoverflow.com/a/27912352 suggests that frompyfunc is a way of writing a generalized accumulate. I don't expect big time savings, maybe 2x.

---

To use frompyfunc, define:

deffoo(x,y):return10*x+y

The loop application (above) would be

def loopfoo(upgain,u,u1):
    for x in upgain:
        u1=foo(u1,x)
        u.append(u1)
    return u

The 'vectorized' version would be:

vfoo=np.frompyfunc(foo,2,1) # 2 in arg, 1 out
vfoo.accumulate(upgain,dtype=object).astype(float)

The dtype=object requirement was noted in the prior SO, and https://github.com/numpy/numpy/issues/4155

In[1195]: loopfoo([1,.1,.2,.3,.4],[],0)
Out[1195]: [1, 10.1, 101.2, 1012.3, 10123.4]In[1196]: vfoo.accumulate([1,.1,.2,.3,.4],dtype=object)
Out[1196]: array([1.0, 10.1, 101.2, 1012.3, 10123.4], dtype=object)

For this small list, loopfoo is faster (3µs v 21µs)

For a 100 element array, e.g. biggain=np.linspace(.1,1,100), the vfoo.accumulate is faster:

In [1199]:timeitloopfoo(biggain,[],0)1000 loops,best of 3:281µsperloopIn [1200]:timeitvfoo.accumulate(biggain,dtype=object)10000loops,best of 3:57.4µsperloop

For an even larger biggain=np.linspace(.001,.01,1000) (smaller number to avoid overflow), the 5x speed ratio remains.