*

Alej

  • ***
  • 78 posts
Configuring the robots.txt file
« on: March 19, 2025, 03:52:48 PM »
Hi everyone, I'm configuring the robots.txt file for my classifieds website, but I don't really know what I should block and what I shouldn't. I'm currently using: "User-agent: *
Disallow: /" to disable all indexing.

Instead of disabling everything, I would like to disable only the following data:

- Users
- /oc-admin
- All search and filter pages (for example: /search/pattern, car)
- index.php?page=ajax&action=runhook&hook=ur_ajax_list&userId=1&itemId= (I don't know why this page was indexed)
- All sections and files of osclass that shouldn't be indexed (I don't know).


To index only the following:

- User posts
- Categories
- Locations
- Static pages (Login, Register, Contact, etc.)


I would really appreciate it if someone could help me create the file, as I don't know much about osclass.
Thanks.
« Last Edit: March 19, 2025, 05:39:28 PM by Alej »

Re: Configuring the robots.txt file
« Reply #1 on: March 19, 2025, 09:40:01 PM »
Hello Alej, try this:

User-agent: *
Disallow: /oc-admin/
Disallow: /user/
Disallow: /search/
Disallow: /index.php?page=ajax&action=runhook&hook=ur_ajax_list&userId=*
Disallow: /*?page=ajax*
Disallow: /*&action=*
Disallow: /*&hook=*
Disallow: /*&userId=*
Disallow: /*&itemId=*

# Allow indexing of important pages
Allow: /category/
Allow: /location/
Allow: /post/
Allow: /static/
Allow: /contact/
Allow: /login/
Allow: /register/

Sitemap: https://yourwebsite.com/sitemap.xml

*

Alej

  • ***
  • 78 posts
Re: Configuring the robots.txt file
« Reply #2 on: March 20, 2025, 03:34:09 PM »
Shouldn't I also block the osclass files? Should I add anything else? Could @MB Themes confirm if the settings are correct, or what I should put in robots.txt?

*

MB Themes

Re: Configuring the robots.txt file
« Reply #3 on: March 22, 2025, 08:13:03 AM »
Default robots.txt that id in install folder should be sufficient
  To get fast support, we need following details: Detail description, URL to reproduce problem, Screenshots

*

Alej

  • ***
  • 78 posts
Re: Configuring the robots.txt file
« Reply #4 on: March 22, 2025, 03:53:58 PM »
When I installed osclass and the epsilon theme, for some reason the robots.txt file was not created. Could you tell me the default content of it?

Re: Configuring the robots.txt file
« Reply #5 on: March 23, 2025, 05:57:22 PM »
User-agent: *
Disallow: /oc-content/downloads/
Disallow: /oc-content/languages/
Disallow: /oc-content/uploads/HTML/
Disallow: /oc-content/uploads/minify/
Disallow: /oc-content/uploads/temp/
Disallow: /oc-includes/osclass/
Disallow: /oc-includes/vendor/

*

Alej

  • ***
  • 78 posts
Re: Configuring the robots.txt file
« Reply #6 on: March 26, 2025, 12:31:24 AM »
Thank you!