How To Merge On Approximate Strings?

I would like to merge 2 Data Frames on approximate country name with merge as of but i am getting the following error: TypeError:'NoneType' object is not callable Please see the il

Solution 1:

This should get you close but it will not be 100% accurate. You can use fuzzywuzzy. fuzzywuzzy uses Levenshtein distance to calculate the difference between two strings:

from fuzzywuzzy import process

# create a choice list
choices = clist['Country'].values.tolist()

# apply fuzzywuzzy to each row using lambda expression
cdata['Close Country'] = cdata['Country'].apply(lambda x: process.extractOne(x, choices)[0])

# merge
cdata.merge(clist, left_on='Close Country', right_on='Country')


  Country_x Close Country Country_y  BL?
0    Braizl        Brazil    Brazil   No
1     Rusia        Russia    Russia  Yes
2        us            US        US   No

You can even return the percent match and only keep values > n if you want to only keep matches say greater than 85%

add percent match

from fuzzywuzzy import process

# create a choice list
choices = clist['Country'].values.tolist()

# apply fuzzywuzzy toeachrowusing lambda expression
cdata['Close Country'] = cdata['Country'].apply(lambda x: process.extractOne(x, choices))

# addpercentmatch wiht apply
cdata[['Close Country', 'Percent Match']] = cdata['Close Country'].apply(pd.Series)

# merge
cdata.merge(clist, left_on='Close Country', right_on='Country')

  Country_x Close Country  PercentMatch Country_y  BL?
0    Braizl        Brazil             83    Brazil   No1     Rusia        Russia             91    Russia  Yes
2        us            US            100        US   No

you can do boolean indexing either before the merge to remove bad match then merge:

cdata[['Close Country', 'Percent Match']] = cdata['Close Country'].apply(pd.Series)
cdata = cdata[cdata['Percent Match']>85]

or you can do it after the merge:

merge= cdata.merge(clist, left_on='Close Country', right_on='Country')
merge[merge['Percent Match'] >85]

fuzzywuzzy returns the percent match as part of the process function. In the first example I removed it by calling the first element of the tuple: process.extractOne(x, choices)[0]

Solution 2:

Given your example I came to a solution. This is not very pythonic but it works! (assuming you have a matching country name in clist for every cdata mispelled country)

defget_closest(x, column):
    tmp = 1000for i2, r2 in clist.iterrows():
        levenshtein = editdistance.eval(x,r2['Country'])
        if levenshtein <= tmp:
            tmp = levenshtein
            res = r2

    return res['BL?']

cdata['BL'] = cdata['Country'].apply(lambda x: get_closest(x, clist))

Output :

CountryBL0BraizlNo1usNo2RusiaYes

I am using the editdistance library to compute the levenshtein distance. You can install it with pip :

pip install editdistance