Improve service for fetching data

Created on 9 September 2024, 5 months ago

There have been several issues over the years caused by outdated or incomplete data stored in the different registries. Depending on which registry you use, you may get different results.

In the early days of this module (~Feb 2009) the registry was made selectable mainly to reduce the time it took to populate the database. Timeout happened when you spent a long time with an open ftp and database connection, downloading things from a long distance. Since then there have been changes both to this module and to the typical processing power and networks speeds available, and now database population time is not really an issue.

Instead, the bigger issue has become the difference in the data sets served by the 5 different registries. Some registries only serve complete data for their own registry. Some don't serve the latest data for others. Some use a non-standard naming convention for the files. As a result, Ip2Country has had to make many exceptions when fetching this data, and now our tests are failing because one registry has stopped serving some data it used to serve.

In fact, currently only two out of the five registries (APNIC and RIPE) seem to currently serve up a complete set of the latest data. Using one of the other three registries (AFRINIC, ARIN, LACNIC) result in an incomplete and sometimes outdated set of data.

For some time I have been thinking of refactoring the code so that it gets the data for each registry directly from that registry, instead of getting all five sets of data from one chosen registry. This should provide better data but also may be less reliable because there would then be five points of failure (needing to successfully ftp file from five different locations) instead of just one. That's what I'm going to do in this issue.

πŸ“Œ Task
Status

Fixed

Version

2.1

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States tr Cascadia

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Production build 0.71.5 2024