Contrib.social

Remove profiles folder from robots.txt

Open on Drupal.org →
Created on 29 February 2016, over 9 years ago
Updated 9 July 2025, 6 days ago

Follow-up to #2364343: Fix robots.txt to allow search engines access to CSS, JavaScript and image files →

Problem/Motivation

In #2364343: Fix robots.txt to allow search engines access to CSS, JavaScript and image files → we've fixed the blocking of web resources (css, js and images) by robots.txt. As proposed in #79 → , we should discuss whether or not to remove the profiles folder from robots.txt.

The reason we might want to allow access to the profiles folder is based on the fact that /modules, /themes, /vendor and /sites are not blocked. So why shouldn't we allow profile as well?

Additionally, we can decide later if these folders can be removed from blocking in Drupal 7 as well.

Proposed resolution

Change robots.txt file

Remaining tasks

Decide if the profiles folder can be removed from robots.txt.

User interface changes

none

API changes

none

📌 Task
Status

Postponed: needs info

Version

11.0 🔥

Component

other

Created by

🇳🇱Netherlands neograph734 Netherlands

Live updates comments and jobs are added and updated live.
  • Needs backport to D7

    After being applied to the 8.x branch, it should be considered for backport to the 7.x branch. Note: This tag should generally remain even after the backport has been written, approved, and committed.

  • stale-issue-cleanup

    To track issues in the developing policy for closing stale issues, [Policy, no patch] closing older issues

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • Comment 6 days ago →
    🇺🇸United States smustgrave

    Thank you for creating this issue to improve Drupal.

    We are working to decide if this task is still relevant to a currently supported version of Drupal. There hasn't been any discussion here for over 8 years which suggests that this has either been implemented or is no longer relevant. Your thoughts on this will allow a decision to be made.

    Since we need more information to move forward with this issue, the status is now Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.

    Thanks!

contrib.social Blog FAQ Discussions
Production build 0.71.5 2024