*

Post Free Online

  • ***
  • 34 posts
  • Welcome to Post Free Online classified web site
robots.txt configuration
« on: March 07, 2024, 03:31:37 AM »
Below is the default settings for robots.txt.  All are Disallowed.  What we need to allow to get the search engine crawl the web site.

User-agent: *
Disallow: /oc-content/downloads/
Disallow: /oc-content/languages/
Disallow: /oc-content/uploads/HTML/
Disallow: /oc-content/uploads/minify/
Disallow: /oc-content/uploads/temp/
Disallow: /oc-includes/osclass/
Disallow: /oc-includes/vendor/

Marked as best answer by postfreeonline2 on March 07, 2024, 05:48:12 PM
*

MB Themes

Re: robots.txt configuration
« Reply #1 on: March 07, 2024, 03:56:24 PM »
Only specified folder paths should not be crawled by robots
  To get fast support, we need following details: Detail description, URL to reproduce problem, Screenshots

*

Post Free Online

  • ***
  • 34 posts
  • Welcome to Post Free Online classified web site
Re: robots.txt configuration
« Reply #2 on: April 13, 2024, 08:13:30 AM »
Only specified folder paths should not be crawled by robots

Do we need to mention the allow paths too?
User-agent: Googlebot, bingbot

*

MB Themes

Re: robots.txt configuration
« Reply #3 on: April 13, 2024, 12:11:26 PM »
If you allow, you must define everything you allow.
Alternatively you just disallowwhat you want to keep excluded on google
  To get fast support, we need following details: Detail description, URL to reproduce problem, Screenshots