3

Is there any software library or framework that allows merging data from a different source on regular basis? As I understand it should store and support a different kind of rules for cleaning and reformatting data from a particular source as it is the only way from my point of view.

Exmaple:

Source 1: New Great Sneakers NGS1
Source 2: ngs1 sneakers

So it should be merged after applying rules: convert to lower case -> find some pattern 'NGS*' -> be sure that a number after pattern 'NGS' is the same.

It is some kind of naive way to solve the problem. Is there a better way to perform such task on regular basis?

sphinks
  • 131
  • 2

3 Answers3

1

This can be easily done using Python. What you want to do is very frequently done by Data Engineers and Data Scientists and is known as ETL Pipelines (Extract Transform Load). Basically you have to gather data from different sources, transform them (clean and merge in your case) and load them into one database for further use.

This can be easily done using Pandas which is a data manipulation library in Python. Also use regex (another python library) to find patterns in the string or text.

For extracting data from different sources, there are libraries available in python too. For example you can use web scrapping libraries such as Autoscrapper or Scrappy to scrap data from websites is that is your sources.

If you want to extract data from a database (structured or unstructured) then you have libraries like pymysql (for SQL databases) or pymongo (for MongoDB).

Also you can automate the whole process using python too. There are automation libraries which can automate the whole ETL pipeline.

Cheers!

spectre
  • 2,223
  • 2
  • 14
  • 37
0

That problem is commonly called record linkage.

There are many approaches and software options for record linkage. In your example, it can be done with conditional logic and regular expressions.

Brian Spiering
  • 23,131
  • 2
  • 29
  • 113
0

When you deal with similar strings which need to be merged, you can look at string distance metrics.

There are a number of packages/libraries, e.g for R or Python, which deal with merging similar strings. However, I think there is no off the shelf solution which does fuzzy string matching or record linkage in a (reliable) fully automatic way.

Peter
  • 7,896
  • 5
  • 23
  • 50