Webscraper tool requires scrapingbot module but can be used with simple crawler

Created on 28 March 2025, 15 days ago

Problem/Motivation

The Drupal\ai_agents\Plugin\AiFunctionCall\Webscraper function uses the included WebScraperTrait to scrape a webpage, which in turn relies on EITHER the scrapingbot or simple_crawler modules to do the actual scraping. Howdver, on a site with only simple_crawler enabled the WebScraper cannot be used because it lists a dependency on the scrapingbot module.

Proposed resolution

The issue seems to be that the underlying code that consumes the module_dependencies array is not flexible enough to handle either/or dependencies. Ideally we would update the code so it can do this - possibly by listing OR dependencies within the array key with a || splitter. But if this is not desirable, the dependency on scrapingbot should be removed.

🐛 Bug report
Status

Active

Version

1.1

Component

Code

Created by

🇬🇧United Kingdom MrDaleSmith

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Production build 0.71.5 2024