Best way to clean Belgian cities data in English, Dutch, and French, with random numbers/symbols, spaces, typos in between?

I'm a bit stumped.

As the title says, I have a vector of Belgian cities. It's a mix of English, Dutch, and Belgian way of calling the cities with accents/umlauts/etc., and (probably) some postal codes. Some might be corrupted (I see cents and very few other random symbols).

Setting aside the postal code problem for now, I have some simple regex (e.g. str_replace_all()) that removes spaces, punctuation or few other symbols, and few manually translated (e.g. Bruxelles -> Brussels). It still results in 7.519 unique cities.

First, is there data of Belgian cities in three columns of English, French, and Dutch somewhere?

Would you say this first step is the correct approach? How would you go about this problem? Would fuzzy matching be able to also return 'no match' instead of trying to force a match?

Oh dear. If you are in a large university or a Gov't organization, I'd talk to your translation people. Other then that, you might want to see what the [International Organization for Standardization] (International Organization for Standardization - Wikipedia) (ISO))
can do. See ISO 3166-2 for example but I do not know it they go to the level of cities. Another bet is the International Postal Union. I would think that they would have something like this.

Otherwise, can you post a sample of the data here? A handy way to supply some sample data is the dput() function. In the case of a large dataset something like dput(head(mydata, 100)) should supply the data we need as long as it includes some of the weird coding.

1 Like

Thanks, IPU seems interesting.

can you post a sample of the data here?

I put the lowest occurring cities first, because errors make them rarer, and changed numbers to 123. So here are 100 examples.

df |> count.(city) |> arrange.(-desc(N)) |> head(100) |> select.(city) |> dput()

structure(list(city = c("CROISY SUR EURE", "NUKERKE-MAARKEDAL", 
"HERMIGNIES", "PERONNES LEZ BINCHE", "SIRAUL.T", "ESCH-SUR-ALZETTE-", 
"SAINT-MARTIN-D'HERES", "HAUTMONT", "CARVIN", "DS BREDA", "H?L?CINE", 
"STASSEN", "BETTANGE-SUR-MESS", "CORROY LE CHATEAU", "WANDHAFF", 
"DESSELGHEM", "BG AMERSFOORT", "HERQUEGIES", "JUMET (CHALEROI)", 
"PI?TON", "ESCH S/ALZETTE LUXEMB", "PETITE FORET", "AUGNY", "FĂ‚CHES THUMESNIL", 
"MASSY", "BEZONS", "SAINT-AUBAIN", "LUXEMBOURG ROLLINGERGRUND", 
"VOROUX LEZ LIERS", "DRÉHANCE", "GROBBENCONK", "LEUVEN-HEVERLEE", 
"FLÔNE", "EISENBORN", "PETANGE LAMADELAINE", "COLMAR-BERG", 
"WALDHOF", "HOSTERT-NIEDERANVEN", "WALLENDORF", "FINDEL-LUXEMBOURG", 
"LUXEMBOURG-FINDEL", "WALTENSCHWIL", "MERELBEKE (MELSEN)", "RIJKHOVEN BILZEN", 
"LO-POLLINKHOVE", "PELT (OVERPELT)", "12345/ VIEUX-RENG", "12345/ JEUMONT", 
"LEBBEKE-DENDERBELLE", "SINT-DENYS WESTREM", "LAPSCHEURE-DAMME", 
"BRUGGE / ST.KRUIS", "BRUXELLES - SAINT-GIL", "12345 PADENBORRE", 
"OTTIGNIES - LOUVAIN-", "GERAADSBERGEN (MOERBE", "KOKSIJDE/LV.OOSTDUINK", 
"12345 EDYMM", "VISE/LV.VISE", "BRAINE LE COMPTE", "ONZE-LIEVE-VROUW-WAVE", 
"12345 MAZURY BOUCHER", "12345 MAZARIN", "12345 LIMPERIAL FIS", 
"SAINT - AMAND", "MONTMEDY/LIVR 1234 SA", "MONTEGNEE / LIVR 1234", 
"SERAING / 1234 AN", "STROMBEEK", "1234 COKELAERE - LI", "OUGREE / 1234 LIEGE", 
"FOURON LE COMTE", "DOTTINGIES", "WESPELAAR - HAACHT", "ST.PIETERS LEEUW.", 
"TIENEN/LV.1234 JODOIG", "KALMTHOUT ACHTERBROEK", 
"BOOISCHOT PIJPELHEIDE", "BEGIJNENDIJK BETEKOM", "SINT-TRUIDEN DURAS", 
"SCANNING", "LIEGE ROBERMONT", "PETIT RECHAIN", "JALHAY.", "XHENDELESSE.", 
"JEMEPPES SUR SAMBRE", "SAINT-GEORGES-SUR-ME", "MONTIGNIES S/SAMBRE", 
"ST ANDRE", "LICHTENBUSCH", "HERGERSBERG", "APPELTERRE_EICHEM", 
"ST-MARIA-LIERDE", "BRUSSEL ETTERBEEK", "OUDEGEM/DENDERMONDE", 
"HERDERSEM (AALST)", "AFLIGEM", "ST_KATHERINA_LOMBEEK", "SINT_AGATHA_BERCHEM"
)), row.names = c(NA, -100L), class = c("tidytable", "data.table", 
"data.frame"), .internal.selfref = <pointer: 0x5638e057baa0>)

For some reason' I am getting an error on that dput() data so I simply read it in as a vector.

It is a mess and clearly beyond my skill level. There are some real experts here who should be able to help. Sorry to not be of more help.

1 Like

pathos/jrkrideau
dput() wont work well on a data.table (which is a complex object designed for inplace mutation etc., hence the weirdness with reference/pointers) , therefore use as.data.frame()on it first to make it conventional before passing to dput()

1 Like

Hello,

Do you only have the city names, is the postal code also available?
Postal codes might be your best bet here because if you have them, they can be linked to cities (there are list online that link postal codes to cities)

Using a non standardized address is always a very messy business, and the "easiest" way to use them is to use and API that can read messy addressed and returns the closest standardized address.

An API like the one below might help for example

Note that there is no guarantee that it always will be correct and you also have to pay as soon as you want to do a large number of calls.

Hope this helps,
PJ

1 Like

That was my thought but I do not know enough about R file structures to be sure.

This gives us the vector, horrible though it is. :slight_smile:

city = c("CROISY SUR EURE", "NUKERKE-MAARKEDAL", 
                                  "HERMIGNIES", "PERONNES LEZ BINCHE", "SIRAUL.T", "ESCH-SUR-ALZETTE-", 
                                  "SAINT-MARTIN-D'HERES", "HAUTMONT", "CARVIN", "DS BREDA", "H?L?CINE", 
                                  "STASSEN", "BETTANGE-SUR-MESS", "CORROY LE CHATEAU", "WANDHAFF", 
                                  "DESSELGHEM", "BG AMERSFOORT", "HERQUEGIES", "JUMET (CHALEROI)", 
                                  "PI?TON", "ESCH S/ALZETTE LUXEMB", "PETITE FORET", "AUGNY", "FĂ‚CHES THUMESNIL", 
                                  "MASSY", "BEZONS", "SAINT-AUBAIN", "LUXEMBOURG ROLLINGERGRUND", 
                                  "VOROUX LEZ LIERS", "DRÉHANCE", "GROBBENCONK", "LEUVEN-HEVERLEE", 
                                  "FLÔNE", "EISENBORN", "PETANGE LAMADELAINE", "COLMAR-BERG", 
                                  "WALDHOF", "HOSTERT-NIEDERANVEN", "WALLENDORF", "FINDEL-LUXEMBOURG", 
                                  "LUXEMBOURG-FINDEL", "WALTENSCHWIL", "MERELBEKE (MELSEN)", "RIJKHOVEN BILZEN", 
                                  "LO-POLLINKHOVE", "PELT (OVERPELT)", "12345/ VIEUX-RENG", "12345/ JEUMONT", 
                                  "LEBBEKE-DENDERBELLE", "SINT-DENYS WESTREM", "LAPSCHEURE-DAMME", 
                                  "BRUGGE / ST.KRUIS", "BRUXELLES - SAINT-GIL", "12345 PADENBORRE", 
                                  "OTTIGNIES - LOUVAIN-", "GERAADSBERGEN (MOERBE", "KOKSIJDE/LV.OOSTDUINK", 
                                  "12345 EDYMM", "VISE/LV.VISE", "BRAINE LE COMPTE", "ONZE-LIEVE-VROUW-WAVE", 
                                  "12345 MAZURY BOUCHER", "12345 MAZARIN", "12345 LIMPERIAL FIS", 
                                  "SAINT - AMAND", "MONTMEDY/LIVR 1234 SA", "MONTEGNEE / LIVR 1234", 
                                  "SERAING / 1234 AN", "STROMBEEK", "1234 COKELAERE - LI", "OUGREE / 1234 LIEGE", 
                                  "FOURON LE COMTE", "DOTTINGIES", "WESPELAAR - HAACHT", "ST.PIETERS LEEUW.", 
                                  "TIENEN/LV.1234 JODOIG", "KALMTHOUT ACHTERBROEK", 
                                  "BOOISCHOT PIJPELHEIDE", "BEGIJNENDIJK BETEKOM", "SINT-TRUIDEN DURAS", 
                                  "SCANNING", "LIEGE ROBERMONT", "PETIT RECHAIN", "JALHAY.", "XHENDELESSE.", city = c("CROISY SUR EURE", "NUKERKE-MAARKEDAL", 
                                  "HERMIGNIES", "PERONNES LEZ BINCHE", "SIRAUL.T", "ESCH-SUR-ALZETTE-", 
                                  "SAINT-MARTIN-D'HERES", "HAUTMONT", "CARVIN", "DS BREDA", "H?L?CINE", 
                                  "STASSEN", "BETTANGE-SUR-MESS", "CORROY LE CHATEAU", "WANDHAFF", 
                                  "DESSELGHEM", "BG AMERSFOORT", "HERQUEGIES", "JUMET (CHALEROI)", 
                                  "PI?TON", "ESCH S/ALZETTE LUXEMB", "PETITE FORET", "AUGNY", "FĂ‚CHES THUMESNIL", 
                                  "MASSY", "BEZONS", "SAINT-AUBAIN", "LUXEMBOURG ROLLINGERGRUND", 
                                  "VOROUX Lcity = c("CROISY SUR EURE", "NUKERKE-MAARKEDAL", 
                                  "HERMIGNIES", "PERONNES LEZ BINCHE", "SIRAUL.T", "ESCH-SUR-ALZETTE-", 
                                  "SAINT-MARTIN-D'HERES", "HAUTMONT", "CARVIN", "DS BREDA", "H?L?CINE", 
                                  "STASSEN", "BETTANGE-SUR-MESS", "CORROY LE CHATEAU", "WANDHAFF", 
                                  "DESSELGHEM", "BG AMERSFOORT", "HERQUEGIES", "JUMET (CHALEROI)", 
                                  "PI?TON", "ESCH S/ALZETTE LUXEMB", "PETITE FORET", "AUGNY", "FĂ‚CHES THUMESNIL", 
                                  "MASSY", "BEZONS", "SAINT-AUBAIN", "LUXEMBOURG ROLLINGERGRUND", 
                                  "VOROUX LEZ LIERS", "DRÉHANCE", "GROBBENCONK", "LEUVEN-HEVERLEE", 
                                  "FLÔNE", "EISENBORN", "PETANGE LAMADELAINE", "COLMAR-BERG", 
                                  "WALDHOF", "HOSTERT-NIEDERANVEN", "WALLENDORF", "FINDEL-LUXEMBOURG", 
                                  "LUXEMBOURG-FINDEL", "WALTENSCHWIL", "MERELBEKE (MELSEN)", "RIJKHOVEN BILZEN", 
                                  "LO-POLLINKHOVE", "PELT (OVERPELT)", "12345/ VIEUX-RENG", "12345/ JEUMONT", 
                                  "LEBBEKE-DENDERBELLE", "SINT-DENYS WESTREM", "LAPSCHEURE-DAMME", 
                                  "BRUGGE / ST.KRUIS", "BRUXELLES - SAINT-GIL", "12345 PADENBORRE", 
                                  "OTTIGNIES - LOUVAIN-", "GERAADSBERGEN (MOERBE", "KOKSIJDE/LV.OOSTDUINK", 
                                  "12345 EDYMM", "VISE/LV.VISE", "BRAINE LE COMPTE", "ONZE-LIEVE-VROUW-WAVE", 
                                  "12345 MAZURY BOUCHER", "12345 MAZARIN", "12345 LIMPERIAL FIS", 
                                  "SAINT - AMAND", "MONTMEDY/LIVR 1234 SA", "MONTEGNEE / LIVR 1234", 
                                  "SERAING / 1234 AN", "STROMBEEK", "1234 COKELAERE - LI", "OUGREE / 1234 LIEGE", 
                                  "FOURON LE COMTE", "DOTTINGIES", "WESPELAAR - HAACHT", "ST.PIETERS LEEUW.", 
                                  "TIENEN/LV.1234 JODOIG", "KALMTHOUT ACHTERBROEK", 
                                  "BOOISCHOT PIJPELHEIDE", "BEGIJNENDIJK BETEKOM", "SINT-TRUIDEN DURAS", city = c("CROISY SUR EURE", "NUKERKE-MAARKEDAL", 
                                  "HERMIGNIES", "PERONNES LEZ BINCHE", "SIRAUL.T", "ESCH-SUR-ALZETTE-", 
                                  "SAINT-MARTIN-D'HERES", "HAUTMONT", "CARVIN", "DS BREDA", "H?L?CINE", 
                                  "STASSEN", "BETTANGE-SUR-MESS", "CORROY LE CHATEAU", "WANDHAFF", 
                                  "DESSELGHEM", "BG AMERSFOORT", "HERQUEGIES", "JUMET (CHALEROI)", 
                                  "PI?TON", "ESCH S/ALZETTE LUXEMB", "PETITE FORET", "AUGNY", "FĂ‚CHES THUMESNIL", 
                                  "MASSY", "BEZONS", "SAINT-AUBAIN", "LUXEMBOURG ROLLINGERGRUND", 
                                  "VOROUX LEZ LIERS", "DRÉHANCE", "GROBBENCONK", "LEUVEN-HEVERLEE", 
                                  "FLÔNE", "EISENBORN", "PETANGE LAMADELAINE", "COLMAR-BERG", 
                                  "WALDHOF", "HOSTERT-NIEDERANVEN", "WALLENDORF", "FINDEL-LUXEMBOURG", 
                                  "LUXEMBOURG-FINDEL", "WALTENSCHWIL", "MERELBEKE (MELSEN)", "RIJKHOVEN BILZEN", 
                                  "LO-POLLINKHOVE", "PELT (OVERPELT)", "12345/ VIEUX-RENG", "12345/ JEUMONT", 
                                  "LEBBEKE-DENDERBELLE", "SINT-DENYS WESTREM", "LAPSCHEURE-DAMME", 
                                  "BRUGGE / ST.KRUIS", "BRUXELLES - SAINT-GIL", "12345 PADENBORRE", 
                                  "OTTIGNIES - LOUVAIN-", "GERAADSBERGEN (MOERBE", "KOKSIJDE/LV.OOSTDUINK", 
                                  "12345 EDYMM", "VISE/LV.VISE", "BRAINE LE COMPTE", "ONZE-LIEVE-VROUW-WAVE", 
                                  "12345 MAZURY BOUCHER", "12345 MAZARIN", "12345 LIMPERIAL FIS", 
                                  "SAINT - AMAND", "MONTMEDY/LIVR 1234 SA", "MONTEGNEE / LIVR 1234", 
                                  "SERAING / 1234 AN", "STROMBEEK", "1234 COKELAERE - LI", "OUGREE / 1234 LIEGE", 
                                  "FOURON LE COMTE", "DOTTINGIES", "WESPELAAR - HAACHT", "ST.PIETERS LEEUW.", 
                                  "TIENEN/LV.1234 JODOIG", "KALMTHOUT ACHTERBROEK", 
                                  "BOOISCHOT PIJPELHEIDE", "BEGIJNENDIJK BETEKOM", "SINT-TRUIDEN DURAS", 
                                  "SCANNING", "LIEGE ROBERMONT", "PETIT RECHAIN", "JALHAY.", "XHENDELESSE.", 
                                  "JEMEPPES SUR SAMBRE", "SAINT-GEORGES-SUR-ME", "MONTIGNIES S/SAMBRE", 
                                  "ST ANDRE", "LICHTENBUSCH", "HERGERSBERG", "APPELTERRE_EICHEM", 
                                  "ST-MARIA-LIERDE", "BRUSSEL ETTERBEEK", "OUDEGEM/DENDERMONDE", 
                                  "HERDERSEM (AALST)", "AFLIGEM", "ST_KATHERINA_LOMBEEK", "SINT_AGATHA_BERCHEM"
                                  "SCANNING", "LIEGE ROBERMONT", "PETIT RECHAIN", "JALHAY.", "XHENDELESSE.", 
                                  "JEMEPPES SUR SAMBRE", "SAINT-GEORGES-SUR-ME", "MONTIGNIES S/SAMBRE", 
                                  "ST ANDRE", "LICHTENBUSCH", "HERGERSBERG", "APPELTERRE_EICHEM", 
                                  "ST-MARIA-LIERDE", "BRUSSEL ETTERBEEK", "OUDEGEM/DENDERMONDE", 
                                  "HERDERSEM (AALST)", "AFLIGEM", "ST_KATHERINA_LOMBEEK", "SINT_AGATHA_BERCHEM"EZ LIERS", "DRÉHANCE", "GROBBENCONK", "LEUVEN-HEVERLEE", 
                                  "FLÔNE", "EISENBORN", "PETANGE LAMADELAINE", "COLMAR-BERG", 
                                  "WALDHOF", "HOSTERT-NIEDERANVEN", "WALLENDORF", "FINDEL-LUXEMBOURG", 
                                  "LUXEMBOURG-FINDEL", "WALTENSCHWIL", "MERELBEKE (MELSEN)", "RIJKHOVEN BILZEN", 
                                  "LO-POLLINKHOVE", "PELT (OVERPELT)", "12345/ VIEUX-RENG", "12345/ JEUMONT", 
                                  "LEBBEKE-DENDERBELLE", "SINT-DENYS WESTREM", "LAPSCHEURE-DAMME", 
                                  "BRUGGE / ST.KRUIS", "BRUXcity = c("CROISY SUR EURE", "NUKERKE-MAARKEDAL", 
                                  "HERMIGNIES", "PERONNES LEZ BINCHE", "SIRAUL.T", "ESCH-SUR-ALZETTE-", 
                                  "SAINT-MARTIN-D'HERES", "HAUTMONT", "CARVIN", "DS BREDA", "H?L?CINE", 
                                  "STASSEN", "BETTANGE-SUR-MESS", "CORROY LE CHATEAU", "WANDHAFF", 
                                  "DESSELGHEM", "BG AMERSFOORT", "HERQUEGIES", "JUMET (CHALEROI)", 
                                  "PI?TON", "ESCH S/ALZETTE LUXEMB", "PETITE FORET", "AUGNY", "FĂ‚CHES THUMESNIL", 
                                  "MASSY", "BEZONS", "SAINT-AUBAIN", "LUXEMBOURG ROLLINGERGRUND", 
                                  "VOROUX LEZ LIERS", "DRÉHANCE", "GROBBENCONK", "LEUVEN-HEVERLEE", 
                                  "FLÔNE", "EISENBORN", "PETANGE LAMADELAINE", "COLMAR-BERG", 
                                  "WALDHOF", "HOSTERT-NIEDERANVEN", "WALLENDORF", "FINDEL-LUXEMBOURG", 
                                  "LUXEMBOURG-FINDEL", "WALTENSCHWIL", "MERELBEKE (MELSEN)", "RIJKHOVEN BILZEN", 
                                  "LO-POLLINKHOVE", "PELT (OVERPELT)", "12345/ VIEUX-RENG", "12345/ JEUMONT", 
                                  "LEBBEKE-DENDERBELLE", "SINT-DENYS WESTREM", "LAPSCHEURE-DAMME", 
                                  "BRUGGE / ST.KRUIS", "BRUXELLES - SAINT-GIL", "12345 PADENBORRE", 
                                  "OTTIGNIES - LOUVAIN-", "GERAADSBERGEN (MOERBE", "KOKSIJDE/LV.OOSTDUINK", 
                                  "12345 EDYMM", "VISE/LV.VISE", "BRAINE LE COMPTE", "ONZE-LIEVE-VROUW-WAVE", 
                                  "12345 MAZURY BOUCHER", "12345 MAZARIN", "12345 LIMPERIAL FIS", 
                                  "SAINT - AMAND", "MONTMEDY/LIVR 1234 SA", "MONTEGNEE / LIVR 1234", 
                                  "SERAING / 1234 AN", "STROMBEEK", "1234 COKELAERE - LI", "OUGREE / 1234 LIEGE", 
                                  "FOURON LE COMTE", "DOTTINGIES", "WESPELAAR - HAACHT", "ST.PIETERS LEEUW.", 
                                  "TIENEN/LV.1234 JODOIG", "KALMTHOUT ACHTERBROEK", 
                                  "BOOISCHOT PIJPELHEIDE", "BEGIJNENDIJK BETEKOM", "SINT-TRUIDEN DURAS", 
                                  "SCANNING", "LIEGE ROBERMONT", "PETIT RECHAIN", "JALHAY.", "XHENDELESSE.", 
                                  "JEMEPPES SUR SAMBRE", "SAINT-GEORGES-SUR-ME", "MONTIGNIES S/SAMBRE", 
                                  "ST ANDRE", "LICHTENBUSCH", "HERGERSBERG", "APPELTERRE_EICHEM", 
                                  "ST-MARIA-LIERDE", "BRUSSEL ETTERBEEK", "OUDEGEM/DENDERMONDE", 
                                  "HERDERSEM (AALST)", "AFLIGEM", "ST_KATHERINA_LOMBEEK", "SINT_AGATHA_BERCHEM"ELLES - SAINT-GIL", "12345 PADENBORRE", 
                                  "OTTIGNIES - LOUVAIN-", "GERAADSBERGEN (MOERBE", "KOKSIJDE/LV.OOSTDUINK", 
                                  "12345 EDYMM", "VISE/LV.VISE", "BRAINE LE COMPTE", "ONZE-LIEVE-VROUW-WAVE", 
                                  "12345 MAZURY BOUCHER", "12345 MAZARIN", "12345 LIMPERIAL FIS", 
                                  "SAINT - AMAND", "MONTMEDY/LIVR 1234 SA", "MONTEGNEE / LIVR 1234", 
                                  "SERAING / 1234 AN", "STROMBEEK", "1234 COKELAERE - LI", "OUGREE / 1234 LIEGE", 
                                  "FOURON LE COMTE", "DOTTINGIES", "WESPELAAR - HAACHT", "ST.PIETERS LEEUW.", 
                                  "TIENEN/LV.1234 JODOIG", "KALMTHOUT ACHTERBROEK", 
                                  "BOOISCHOT PIJPELHEIDE", "BEGIJNENDIJK BETEKOM", "SINT-TRUIDEN DURAS", 
                                  "SCANNING", "LIEGE ROBERMONT", "PETIT RECHAIN", "JALHAY.", "XHENDELESSE.", 
                                  "JEMEPPES SUR SAMBRE", "SAINT-GEORGES-SUR-ME", "MONTIGNIES S/SAMBRE", 
                                  "ST ANDRE", "LICHTENBUSCH", "HERGERSBERG", "APPELTERRE_EICHEM", 
                                  "ST-MARIA-LIERDE", "BRUSSEL ETTERBEEK", "OUDEGEM/DENDERMONDE", 
                                  "HERDERSEM (AALST)", "AFLIGEM", "ST_KATHERINA_LOMBEEK", "SINT_AGATHA_BERCHEM"
                                  "JEMEPPES SUR SAMBRE", "SAINT-GEORGES-SUR-ME", "MONTIGNIES S/SAMBRE", 
                                  "ST ANDRE", "LICHTENBUSCH", "HERGERSBERG", "APPELTERRE_EICHEM", 
                                  "ST-MARIA-LIERDE", "BRUSSEL ETTERBEEK", "OUDEGEM/DENDERMONDE", 
                                  "HERDERSEM (AALST)", "AFLIGEM", "ST_KATHERINA_LOMBEEK", "SINT_AGATHA_BERCHEM"
1 Like

pieterjanvc
Might work but I suspect the data set that pathos has is historical and the names of cities and provinces, (even countries?) are flexible--a poor term but I cannot think of a better.

1 Like

Ah yes, I can use postal code! Then do fuzzy matching on the cities.
I used this website here:
https://www.geonames.org/export
There is ' Postal codes are available as a separate download' in the link.
Thanks!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.