robots.txt and WMT | Default Themes

Created on 16 September 2014, almost 11 years ago

Updated 30 April 2025, 2 months ago

Problem/Motivation

it only affects those using a default theme
The default robots.txt uses Disallow: /core/ and under /core/ in Drupal8 we have the default Themes. This means a Searchengine can´t crawl the default Themes.

Proposed resolution

Google announced allowing Googlebot to retrieve JavaScript and CSS.

My advice would be to change the robots.txt like this.

User-agent: *
Crawl-delay: 10
# Directories
Disallow: /core/assets
Disallow: /core/includes
Disallow: /core/lib
Disallow: /core/misc
Disallow: /core/modules
Disallow: /core/profiles
Disallow: /core/scripts
Disallow: /core/test
Disallow: /core/vendor
Disallow: /profiles/
#Files in Core Dir
Disallow: /core/authorize.php
Disallow: /core/CHANGELOG.txt
Disallow: /core/core.services.yml
Disallow: /core/INSTALL.mysql.txt
Disallow: /core/INSTALL.pgsql.txt
Disallow: /core/install.php
Disallow: /core/phpunit.xml.dist
Disallow: /core/update.php
Disallow: /core/UPGRADE.txt
Disallow: /core/.gitignore

are you agree?

📌 Task

Status

Postponed: needs info

Version

11.0 🔥

Component

other

Created by

🇩🇪Germany eule

Live updates comments and jobs are added and updated live.

theme

robots.txt

stale-issue-cleanup
To track issues in the developing policy for closing stale issues, [Policy, no patch] closing older issues

Incomplete comments

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Comment 2 months ago →
🇺🇸United States smustgrave
Thank you for creating this issue to improve Drupal.

We are working to decide if this task is still relevant to a currently supported version of Drupal. There hasn't been any discussion here for over 8 years which suggests that this has either been implemented or is no longer relevant. Your thoughts on this will allow a decision to be made.

Since we need more information to move forward with this issue, the status is now Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.

Thanks!

contrib.social Blog FAQ Discussions

Production build 0.71.5 2024