Ignore front end vendor folders to improve directory search performance

Created on 28 August 2014, almost 10 years ago
Updated 12 September 2023, 10 months ago

Problem/Motivation

Drupal recursively scans the front end vendor directories: node_modules and bower_components.

This causes a noticeable and significantly slower response time when front end developers do common tasks such as clearing the cache (#59) or when rebuilding the theme registry: drupal_find_theme_templates():

drush cc all with node_modules in Drupal theme's root folder: 45 secs
drush cc all without node_modules: 10 secs

Also see statistics from XHProf before and after the proposed fixes in comments #119 (D8) and #105 (D7). In Drupal 8, in certain scenarios, there's over a 2x performance improvement for expensive operations like cache clears, and in Drupal 7, over 3x.

These front end developer tools are very common and highly used (even outside of Drupal).

NPM Stats (https://www.npmjs.com):

175,409 total packages
29,619,131 downloads in the last day
490,004,267 downloads in the last week
1,939,777,453 downloads in the last month

Bower Stats via NPM (https://www.npmjs.com/package/bower):

23,318 downloads in the last day
360,264 downloads in the last week
1,474,986 downloads in the last month
38,599 packages (http://bower.io/stats/)

Grunt Stats (https://www.npmjs.com/package/grunt)

19,102 downloads in the last day
318,143 downloads in the last week
1,290,279 downloads in the last month

All these tools are NodeJS oriented and use the node_modules folder, with the addition of the bower_components folder for Bower.

Due to the combined and shared nature of the node_modules folder, the average size of node_modules is roughly ~5,000-10,000 files. This can, on occasion, easily skyrocket to upwards of ~30,000-50,000 files depending on the size of the project and the node modules used.

Suffice it to say, this isn't some "small deal" Drupal can afford to simply ignore.

Can't people just use a different directory other than node_modules?
No. Node has already made it quite clear that they will never let this happen. Also, this would still be considered a "Drupalism" because front end developers new to Drupal wouldn't know about this issue.

Proposed resolution

Add the node_modules and bower_components folders as a default for the "nomask" option in file_scan_directory().

Remaining tasks

  • Update patch to reflect current issue summary

User interface changes

None

API changes

Technically this is considered an API change, however the event of this happening is very highly unlikely. However:

If you installed Drupal themes or modules via npm, you will be required to move them out of npm's default node_modules folder.
If you grouped sub-modules under the directory name "node_modules", you will be required to rename it.

Adding an update hook as mentioned in #57/#58 to warn users about possible existing directories may be the best way to do this?

πŸ“Œ Task
Status

Fixed

Version

8.2 ⚰️

Component
File systemΒ  β†’

Last updated about 16 hours ago

Created by

πŸ‡¨πŸ‡¦Canada afoster

Live updates comments and jobs are added and updated live.
  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

  • Needs backport to D7

    After being applied to the 8.x branch, it should be considered for backport to the 7.x branch. Note: This tag should generally remain even after the backport has been written, approved, and committed.

Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

Production build 0.69.0 2024