Removing non-simple duplicates in Excel. Python library and fuzzywuzzy formula (levenshtein)

Data Parsing, Python — incorrectly specified categories?

Project translated automatically. Log in or register, to view the original

Hello. My programmer and I tried to remove duplicates in the headings. The formula sometimes deletes a normal heading, and sometimes it leaves an extra duplicate. I have attached the code. I have also included a test file. If you can do it better than us, the output file will only contain the headings that I marked in bold. Please write to me directly. Or make changes in your bid. Let me know if you succeeded. Take the code. Improve it as much as possible.
ATTENTION! In the test file. The headings are highlighted in bold. The task is to ensure that they remain in the output file after processing with the formula script. The headings that are not highlighted in bold are duplicates and should be removed. We had different results. We came up with many ideas. But nothing effective came out. For example, Levenshtein consumes many normal headings that are not duplicates. Playing with % similarity is not very effective. It didn't yield anything good. Here, we need to write some good logic. Logic for when and for which headings to apply and what percentage. And do not remove headings with less than 80% similarity from one type of function. This is not enough. Because it will still leave something unnecessary. For example, a heading with many words. It will no longer count as a duplicate. And it will give it 50% similarity. We also thought of ways to bypass this. We added a second function. Like a reverse. The script makes the first pass with one logic of about 80%. The second pass, on the contrary, does not take headings that are less than 50% similarity. This is to remove long headings that are duplicates. We also thought it might get caught on various symbols like , : and so on. Long headings have commas and so on. And we thought that during the third pass, it would apply 50% specifically to such cases. In short, to find a way to address this.

Proposals 2 Discussions 2

Tetyana Shumeyko

73 4

Budget: 2000 UAH Deadline: 2 days

Good day! I have a lot of experience working with ET! I can try to help you with this issue! Feel free to reach out!!!

Aleksandr Kremenyuk
Vinnytsia, Ukraine
Кременюк О.О

Projects 44
Rating 5.0
Rating 644

Removing non-simple duplicates in Excel. Python library and fuzzywuzzy formula (levenshtein)

Tetyana Shumeyko

Proposals concealed

Proposals are currently absent

Current freelance projects in the category Data Parsing

Parsing prices and product relevance

Set up automation for updating product stock on Prom from the supplier's file.

Parsing and unloading

It is necessary to implement a project for collecting and structuring a large array of images.

Product parser + n8n setup with AI processing