I would like a plugin that would run in the background and scan websites for meta data. Example: website link, Title, description and automatically list them into a category.

There was always problem that if target site detevt this behaviour, your server might get blocked and you are done...

I wonder how google and duckduckgo gets away with scrapping websites, just was wondering.

They are considered as robots so you should be able to see in your accesslog your site was crawled by some robot.
But you do not want to do favor for target site, you wants its content