Support Forums - Classified Ads Script Osclass

General osclass questions => Feature request => Topic started by: mheston on February 23, 2023, 07:34:51 PM

Title: Web crawler plugin?
Post by: mheston on February 23, 2023, 07:34:51 PM

I would like a plugin that would run in the background and scan websites for meta data. Example: website link, Title, description and automatically list them into a category.

Title: Re: Web crawler plugin?
Post by: MB Themes on February 23, 2023, 11:05:32 PM

There was always problem that if target site detevt this behaviour, your server might get blocked and you are done...

Title: Re: Web crawler plugin?
Post by: mheston on February 23, 2023, 11:21:36 PM

I wonder how google and duckduckgo gets away with scrapping websites, just was wondering.

Title: Re: Web crawler plugin?
Post by: MB Themes on February 24, 2023, 08:57:51 AM

They are considered as robots so you should be able to see in your accesslog your site was crawled by some robot.
But you do not want to do favor for target site, you wants its content  ;D