Technical task for an aggregator of product feeds in various formats from different suppliers into one feed

Goal

to obtain a unified YML feed with products from various suppliers for further use in online stores (specifically based on Khoroshop).

Technologies

The stack of technologies used is not critical, but there should be the possibility of deployment on cheap shared hosting or a local web server.

Requirements and functional features

1. Login/Registration. Basic login functionality is required using a username (email) and password. New users can only be added by a logged-in user. There is no option for self-registration. The login form is protected via reCAPTCHA/Cloudflare (if the app is not deployed locally).

2. Setting up the output feed. An interface is needed that allows:

   1. To set the structure (template) of the future YML file (with the ability to edit and save this structure).

   2. Also, in this same section of the web application, there should be the ability to set the structure (tree) of product categories, their editing, and saving.

3. Setting up sources / input feeds (suppliers).

   1. We should have the ability to create an arbitrary number of suppliers, and for each of them set a feed.

   2. The feed can be in various formats XML/YML/XLSX.

   3. The feed can be available via a public link and be uploaded automatically, as well as attached manually by the manager (through the web interface).

   4. There should be a preview option for the supplier's feed (in both structural format and table format), for setting up further connections:

      1. To establish a connection between the supplier's categories and the categories of the output file (which are set in p.2.b)

      2. To obtain a list of all attribute names (characteristics) of the product and set a name for each of them in the output feed. (for example: Name, Price, Article, Availability, Description, Brand, Color, Country of origin, …), as well as the ability to set the ignoring of this attribute (not to include the attribute in the resulting feed).

      3. To set a unique product identifier for this feed (id or another attribute, for example “Article”). This field will be used to track duplicates of products from different suppliers.

      4. If the feed is in XLSX format, then all attributes of the product or, for example, all images of the product can be in one cell. Therefore, it is important to foresee the possibility of post-processing such cells and splitting by delimiter. (for example, for images the delimiter is “,” and for characteristics “;”, where the attribute name and its value are separated by the combination “:␣”).

      5. All the above settings for the source (supplier feed) must be saved.

      6. Before each processing of the input feed, there should be a check against the saved settings to detect new attributes for which no connection has been established. If such attributes are found - send a notification to the email addresses of all registered users.

4. Setting up processing.

   1. Based on the saved settings of each source/supplier, a list of all attributes needs to be formed, and if this attribute is repeated, then allow setting the priority of sources. For example, if for a conditional product Description is set with priority: Supplier_2, Supplier_3, Supplier_1, then when forming the result, we check if this Description is present in Supplier_2, if it is - we output it, if not, then we check in Supplier_3, and so on.

   2. The price is formed based on the characteristic “Availability”: among all suppliers where the product is available, we choose the lowest price.

   3. Processing of input feeds from suppliers should be performed in the background, with a specified interval set through the settings (Cronjob).

   4. Processing of input feeds can be started manually at any time.

   5. In case of errors during the result generation process - send an email to all users.