*

mihaipmp

  • *
  • 4 posts
[Error] Missing cities from the latest Romania locations dataset
« on: February 09, 2024, 04:06:19 AM »
Working on a transfer from two OsClass instances, I noticed the difference between the cities list in one and the other. The source site has the list based on the Osclass 3.x locations, the destination site has the latest location list, imported directly - this seems to be the same as the location list for Osclass 4.x.

Although the new list has many more records, as it ads villages, doing a lookup on the slug, over 200 cities from the first list are missing from the new list. Some are listed with a different slug, but I manually checked and found that quite a few of those are actually missing from the current list.

Example (from the 3.x locations RO-Romania.sql):
(462847, 782155, 'RO', 'Campulung', 1, 'campulung'),
This city, municipality in Romania with close to 30.000 population, is missing from the current list.

Other specific examples present in the old version, missing in the current one:
(463117, 782158, 'RO', 'Bistrita Bargaului', 1, 'bistrita-bargaului'),
(463021, 782157, 'RO', 'Avram Iancu', 1, 'avram-iancu'),

Two questions:
1. Is this problem on the developers' radar, with any chance of being fixed in a decent time?

2. Can I replace these locations on my current version of OsClass (8.2.1) with the SQL data from Osclass 3.x locations, or could this create other errors in the system?

*

MB Themes

Re: [Error] Missing cities from the latest Romania locations dataset
« Reply #1 on: February 11, 2024, 10:30:06 AM »
1. Its autogenerated and we only include data based on specific criteria & population

2. Old datasets miss coordinates
  To get fast support, we need following details: Detail description, URL to reproduce problem, Screenshots

*

mihaipmp

  • *
  • 4 posts
Re: [Error] Missing cities from the latest Romania locations dataset
« Reply #2 on: February 11, 2024, 02:59:13 PM »
Something in your criteria is certainly wrong. In the same county (named Region in OsClass) where Campulung is missing, the first "City" is Adunati - which is in fact a village of 144 inhabitants. But you are missing the second largest city from the county - Campulung - municipality, population 27,574.

Also in the same Region - Poienari - what village is this? The village Poienari from the Poienarii de Muscel commune, or the village Poienari from the Poienarii de Arges commune? One village is in the East of the Region, the other is in the West, and there's a 1.5h drive between them.

Most likely, these are not all the problems in this county, and this is only one county in 41 in Romania - from what I saw so far, all of them have problems.

From my point of view, this dataset is unusable in a production site.

If somebody from the OsClass team is available, I can help guide the regeneration of the locations for Romania to fix all these problems. This should be doable in less that one hour, certainly much, much less time than it would take to manually check and fix all the problems in the current dataset.

*

MB Themes

  To get fast support, we need following details: Detail description, URL to reproduce problem, Screenshots

*

mihaipmp

  • *
  • 4 posts
Re: [Error] Missing cities from the latest Romania locations dataset
« Reply #4 on: February 11, 2024, 04:19:50 PM »
I've checked a few of the locations missing from OsClass and all of them have a population of 0 in this dataset - that is certainly an error and the cause of the problem in the OsClass data.

There are also some other errors (eg. wrong diacritics used in some places - ţ = U+0163 instead of ț = U+021B, and others like this). I will crossreference this dataset with another one, that has the correct spelling, and limit the list of locations to NUTS4 (city / town / commune) instead of level 5 (village). I will get back here with the resulting data file.
« Last Edit: February 11, 2024, 10:32:14 PM by mihaipmp »

*

mihaipmp

  • *
  • 4 posts
Re: [Error] Missing cities from the latest Romania locations dataset
« Reply #5 on: February 12, 2024, 12:03:15 AM »
I have worked with the GeoNames data that you have provided, following these steps:
  • filter to keep only the records with feature code = ADM2 (all the municipalities & towns & communes) + those with feature code = ADM3 (the 6 sectors of the capital, Bucuresti)
  • remove Bucuresti from the list (in OsClass it appears as a Region)
  • remove the type of city (Municipiu, Oraș, Comuna) from all the names
  • correct the diacritics
  • generate the ID os_t_city.pk_i_id = 3,000,000 + geonameid
  • rebuild the reference to the regions - fk_i_region_id
  • generate the CSV table with all the fields needed for the import - attached in the zip
  • manual check and correction of all the remaining problems
  • generate the SQL file for import - attached in the zip

The resulting CSV and SQL files have 3186 cities, which corresponds with the number from the Ministry of Administration's website (http://www.dpfbl.mdrap.ro/nr_uat-uri.html): 102 municipalities (103 - Bucuresti) + 216 towns + 2,862 communes + 6 sectors = 3186 cities.

I executed an import of the SQL file in the OsClass admin without any problem.

In my opinion, this data set should replace the default for Romania. Buf feel free to check it for yourself and let me know if you have any questions.

*

MB Themes

Re: [Error] Missing cities from the latest Romania locations dataset
« Reply #6 on: February 12, 2024, 11:26:22 AM »
You can do that, we will not do that.
We only accept ADM3 & ADM4 for cities.

Thanks for sharing ;)
  To get fast support, we need following details: Detail description, URL to reproduce problem, Screenshots