Millions of cache files create issues

I can see from earlier issues that there is already focus on the potentially huge number of files that File Cache can generate.

For for now I solved my problem with a script that simply deletes the oldest files. This seems to have very little influence on the performence of the site, but certainly keeps the number of files in a manageable range.

For whatever it's worth to others, here's my script. I called it delete_oldest.sh:

#!/bin/bash

# Constants
NUM_FILES_TO_DELETE=1000

# Usage function
usage() {
  echo "Usage: $0 [-y] /path/to/directory"
  echo "  -y    Skip confirmation prompt"
  exit 1
}

# Check for arguments
CONFIRM=false
while getopts ":y" opt; do
  case ${opt} in
    y )
      CONFIRM=true
      ;;
    \? )
      usage
      ;;
  esac
done

# Shift out processed options
shift $((OPTIND -1))

# Get directory argument
TARGET_DIR="$1"

if [ -z "$TARGET_DIR" ]; then
  usage
fi

# Make sure the directory exists
if [ ! -d "$TARGET_DIR" ]; then
  echo "Error: Directory '$TARGET_DIR' does not exist."
  exit 1
fi

# Confirm if not using -y
if [ "$CONFIRM" = false ]; then
  read -p "Are you sure you want to delete the $NUM_FILES_TO_DELETE oldest files in '$TARGET_DIR'? [y/N]: " answer
  case "$answer" in
    [yY][eE][sS]|[yY])
      ;;
    *)
      echo "Operation cancelled."
      exit 0
      ;;
  esac
fi

# Delete oldest files
find "$TARGET_DIR" -type f -printf '%T@ %p\n' | \
  sort -n | \
  head -n "$NUM_FILES_TO_DELETE" | \
  cut -d' ' -f2- | \
  xargs -d '\n' rm -f

echo "Deleted the $NUM_FILES_TO_DELETE oldest files in '$TARGET_DIR'"

It's run by cron every 30 minutes for the directories that seems to grow fastest and largest.

*/30 * * * * /usr/bin/env delete_oldest.sh -y /var/filecache/page > /dev/null
*/30 * * * * /usr/bin/env delete_oldest.sh -y /var/filecache/render > /dev/null
*/30 * * * * /usr/bin/env delete_oldest.sh -y /var/filecache/dynamic_page_cache > /dev/null
*/30 * * * * /usr/bin/env delete_oldest.sh -y /var/filecache/data > /dev/null

/var/filecache/ is of course the location of the cache flies, and needs to be adjusted according to your setup.

The script can also run from the command line as
delete_oldest.sh /var/filecache/render

My version deletes the oldest 1000 files, and for now that seems to tame the file numbers. I started out with 10000 to get the number of files reduced, but after a while it seems that 1000 is an OK number to cull.

Martin

This appears to be a duplicate of https://www.drupal.org/project/filecache/issues/3001324 ✨ Limit number of cache files in cache bin directories Needs work . Martin, would you be interested in testing my patch in https://www.drupal.org/project/filecache/issues/3001324#comment-16058304 ✨ Limit number of cache files in cache bin directories Needs work to see if that helps? It splits the cache items into sub-directories within each cache bin directory. We've patched our sites with it, and it has made a dramatic difference. But I don't think any of the sites are as large as the one you describe.

@nubeli,

I did see that comment and thought about using that approach. I'll try the patch and report back. It may solve the problem with handling the colossal number of files.

But it may not solve my basic problem: the disk filling up with huge amounts of cached data in files, which eventually suffocates the site.

I will probably still need some way to handle that problem, and my script seems to do that for now. The site is still very fast, the cache doesn't grow beyond control and it's been running for a few weeks now with no issues.

Martin