For anyone else looking into something like this, I was able to find a workaround solution with this: https://solr.apache.org/guide/6_6/uploading-data-with-solr-cell-using-ap...
Using this command, I got a Solr output object: curl "http://mySolrServer:8983/solr/labor/update/extract?extractOnly=true" --data-binary @/home/ron/workarea/ai/tika/article.pdf -H 'Content-type:application/pdf' > /home/ron/workarea/ai/tika/article.xml
Then, using this php script to read the .xml file, I was able to get the text of the pdf as it is currently stored in the Solr index, but with the linefeeds included.
// Read the JSON response from the XML file
$json = file_get_contents('article.xml');
// Parse the JSON data
$data = json_decode($json, true);
// Extract the XML content from the JSON response
$xml_content = $data[''];
// Create a new DOMDocument object
$dom = new DOMDocument();
// Load the XML content
$dom->loadXML($xml_content);
// Get the <body> element
$body = $dom->getElementsByTagName('body')->item(0);
// Initialize an empty string to store the extracted text
$text_content = '';
// Iterate over the child elements of the <body> element
$elements = $body->childNodes;
foreach ($elements as $element) {
if ($element->nodeType === XML_ELEMENT_NODE) {
// Extract the text content of the element
$text_content .= $element->textContent . "\n";
}
}
// Print or save the extracted text
echo $text_content;
// Or save to a file:
file_put_contents('extracted_text.txt', $text_content);
For comparison, this is the source pdf: https://s3.us-west-2.amazonaws.com/docs.scbbs.com/docs/test/13_Article_1...
And this is the Solr (tika) output: https://www.drupal.org/files/issues/2024-05-18/extracted_text_from_tika.txt →
Still contains page numbers, headers and footers -- but so does the Solr index object.
Any suggestions? Anything?
Thanks much! Will definitely give it a try!
Thank you for the response. I better understand why it works the way it does. But, I still miss being able create the "list of indented paragraph items" like I could before.
I'm going to try the new ckeditor5 plugin pack to see if it helps: https://ckeditor.com/blog/drupal-ckeditor-5-plugin-pack/?utm_source=The+...
If not, I'll try your suggestions.
Again, thanks!
Newlines.
Here is the context: https://community.openai.com/t/using-gpt-4-api-to-semantically-chunk-doc...
I am developing a module called "SolrAI" which essentially facilitates RAG (Retrieval Augmented Generation) on a Drupal site. Search API Solr is a critical component of this module as it creates (via the Solr extractor) the text entities that are sent to the vector store for indexing.
Right now, this text is "chunked" (split into equal segments of x characters|tokens) by simply chopping it sequentially at every x characters.
I would like to introduce "semantic chunking" into my application. Here, the AI model would analyze each text entity, create a json file of it's hierarchy, and "chunk" it based upon that hierarchy.
The problem is, I need to be able to locate the segments within the text to extract. The best way I have found so far to do that (see the discussion in the link above) is by identifying specific lines in the text. This currently is not possible with the text rendered by the Solr extractor because it removes the newline characters.
Now, a workaround that I am thinking of is to use the Solr ID to retrieve the Drupal object (node, file or comment) and process that instead. But this method is prone to error and far less efficient.
This is why I would like to know if I could preserve the newline characters in the Search API Solr extractions.
SomebodySysop → created an issue.
One day, hopefully. I have been working continuously on the module for the past 9 months (unbelievable). Your post motivated me to sit down and create a general list of features and components. I'm still not finished, and I have barely begun to document it. As I have mentioned earlier, I am using it for my own projects right now: https://www.scbbs.com/ai_projects
SolrAI Module Features
All the features available via Drupal plus AI.
Features
Regular Drupal Features:
- Create and maintain nodes files and comments as usual.
- Access Control through Core and:
- Group Module
- Taxonomy Module
SolrAI Module Features (Semantic Search)
- All content automatically inserted into and updated in vector store. Embeddings mirror the Apache Solr database.
- Required:
- Search API Module
- Search API Solr Module
- Search API Attachments Module (for file attachments)
- Weaviate Vector Store Account
- Query any node, file, or comment content on site.
- Content organized by:
- Group Module
- Taxonomy Module
- Headless access through APIs:
- Submit queries via email.
- Submit queries via SMS.
- Submit queries via API.
- User selects content to query.
- User can select from multiple LLMs. Providers currently supported:
- Anthropic
- Mistral
- OpenAI
- Custom user-defined query configurations.
- Templates which determine what libraries will appear and which LLM will be used on individual query screens. Basically custom query screens.
- Custom user-defined embedding configurations.
- Templates which determine embedding schemes. Different content in different groups can be embedded using different methodologies.
- Maintains log of all queries and responses along with associated details.
- Individual users can query and download (CSV file) their query history.
- Requires Views Data Export Module.
- Text to SQL Log Analytics.
SolrAI Physical Components
- 12 Libraries
- 7 Services
- Plugins:
- 3 Queue Plugins
- 5 Block Plugins
- 5 REST Resource Plugins
- 18 Views Plugins
- 16 Query and Settings Forms:
- The primary settings form has some 135 elements.
- 8 Tables:
- Query Log
- Custom Query Configurations
- Custom Embedding Configurations
- User Details
- Temporary Sessions
- 2 Event Subscribers
- 9 Controllers:
- Main Query
- Email Query
- SMS Query
- Group Query
- API Query
- Documentation Query
- Text to SQL Query
- Stripe Processing
- 2 Twig Templates
- 3 CSS Files
- 11 JS Files
- 8 Permissions
Original Video and Written Proposals: https://www.scbbs.com/node/845
Hebrew characters appear to be supported as well as I need them supported. I am sending the text created in my Solr database to be embedded, and I needed the Hebrew characters maintained in UTF-8 encoding, which they appear to be. This allows me to use a large language model to create English translation of Hebrew texts.
FYI, this is what my embeddings look like (for Hebrew).
Search API Solr working exactly as I need it to work -- so far.
Thanks for the response. I have not looked at it in over a year since I don't know enough about the module to develop a patch. It was my hope that someone with much more knowledge would stumble upon this and figure out a solution.
Thank you for the patch.
Managed not used no longer issues error.
Used not managed, however, results in this:
No link template 'canonical' found for the 'paragraph' entity type
Just a note, in case anyone else is looking to do this. I was able to get the search words highlighted in the node text by adding this in the Highlight referrer settings:
^https?:\/\/.+[&?]search_api_fulltext=([^&]+).*$
I also added this to local search patterns:
^https?:\/\/.+\/node\/([0-9]+)$
Not sure which does the trick, but both result in search terms being highlighted in node after I click on it in search results.
SomebodySysop → created an issue.
Update:
I did figure out the workaround:
/topics
Will get topics the beginning of the message. Of course, theoretically it will also get:
/page/topics
But for now, this works for me.
Please let me know if I am correct about the caret (^) issue.
Thank you. I am using REGEXP mode.
According to this page: https://dev.mysql.com/doc/refman/8.0/en/regexp.html#operator_regexp
To match patterns at the start of log entries or any string in SQL using REGEXP, the caret (^) symbol is indeed used to anchor the pattern to the beginning of the string.
What I am saying is that when I enter
^topics
As my pattern, the test fails.
When I enter
topics
The test returns all the log entries that begin with /topics
The problem with using the latter, is that, technically, it will match something like "?topics=", which I DO NOT want matched.
Do you see the issue? I don't think AutoBan REGEXP mode recognizes the caret (^) symbol.
So, in further testing, I discovered that AutoBan REGEXP does not accept regex start end end boundries:
It appears that this is the way to handle the word crawls (no beginning and ending word boundries):
[A-Za-z]+\.([0-9]{1,2})\.([0-9]{1,2})
But, need to know how to match in REGEXP if message starts with a particular pattern, like:
topics
SomebodySysop → created an issue.
SomebodySysop → created an issue.
Yes, we do have at least one paragraph content type configured with file attachment fields.
SomebodySysop → created an issue.
@larowlan
Thank you VERY much for that guidance. Exactly what I needed.
Anyone else need to do something similar, this is how I utilized the Event Subscriber:
namespace Drupal\solrai\EventSubscriber;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Symfony\Component\HttpKernel\KernelEvents;
use Symfony\Component\HttpKernel\Event\ResponseEvent;
use Drupal\Core\Routing\TrustedRedirectResponse;
class SolraiEventSubscriber implements EventSubscriberInterface {
public static function getSubscribedEvents() {
$events[KernelEvents::RESPONSE][] = ['redirectToExternalUrl', 0];
return $events;
}
public function redirectToExternalUrl(ResponseEvent $event) {
$request = $event->getRequest();
if ($request->get('_route') !== 'entity.node.canonical') {
return;
}
$node = $request->get('node');
$nodePath = '/node/' . $node->id();
$externalUrl = $this->lookupPathInTable($nodePath);
if ($externalUrl) {
$redirect = new TrustedRedirectResponse($externalUrl, 301);
$redirect->setMaxAge(2764800);
$event->setResponse($redirect);
}
}
private function lookupPathInTable($path) {
$pathMappings = [
'/node/7849' => 'https://churchages.net/en/sermon/wesley/laying-the-foundation-of-the-new-chapel/',
'/node/7850' => 'https://churchages.net/en/sermon/wesley/death-of-rev-mr-john-fletcher/'
// Add more mappings as needed
];
return $pathMappings[$path] ?? null;
}
}
What I wish to achieve:
Let's say I have a node, /node/210.
I have a table called "source urls". This table contains local paths I want re-directed to external sites. If a match is found, then I would like the user automatically sent to the external site.
So, when someone clicks on a link for this "/node/210" anywhere on my site, they are automatically redirected to the appropriate external site url.
The reason for this: I am building an AI response API that would allow users to embed and deploy a chatbot on their site for their content without them having to build and maintain this infrastructure themselves. So, ideally, /node/210 would consist of data from their site that we have processed for AI interaction, and rather than have the end-user access the data on our site, they are directed to the actual source.
The closest related issue I could find is this: https://www.drupal.org/project/drupal/issues/2873293 🐛 OutboundPathProcessorInterface does not correctly handle absolute urls anymore Closed: works as designed
Which was never resolved.
SomebodySysop → created an issue.
Figured it out. I had the "same IP address" configuration item checked, and I can't use that because the IP address will be the server IP address once user gets to the API controller.
But, the IP address will be their own once they receive link and go to site to confirm.
Duh!
Unchecked and everything works as expected.
So the code here does work: https://www.drupal.org/project/email_confirmer/issues/3019765#comment-12... →
The entity privacy status is changed, so this title is no longer appropriate: How to set entity privacy status to FALSE
Update: Privacy status is being updated.
Field private, Delta 0: {"value":0}
But the links still don't work for anonymous. They do work for logged in users.
SomebodySysop → created an issue.
drops Group 1.x/2.x support
I'm not sure about this, but aren't the majority of people using the Group module using either 1.x or 2.x?
This was my solution to this problem, using hook_views_data_alter(): https://www.drupal.org/project/search_api/issues/3409919 💬 How to create a custom views filter for a seach api solr view? Fixed
Resolved here: https://www.drupal.org/project/search_api/issues/3409919#comment-15374013 💬 How to create a custom views filter for a seach api solr view? Fixed
So, it turns out the resolution in #4 💬 How to create a custom views filter for a seach api solr view? Fixed did in fact work. It turns out that the site I was testing on was using outdated Group labels. The easy fix was to re-index the site, and the Group filter works as it should.
While the SearchApiOptions filter plugin works great for constructing the groups filter, it doesn't filter both nodes and files (even though the files do contain the Solr sbn_add_nodegroup_str field which is created by the Search API processor plugin).
So, I'm thinking the better route is to create my own filter that will modify the Solr query itself. Based upon your suggestions, here is my new search api group node filter class file:
namespace Drupal\sbn\Plugin\views\filter;
use Drupal\search_api\Plugin\views\filter\SearchApiFilter;
use Drupal\views\Plugin\views\display\DisplayPluginBase;
use Drupal\views\ViewExecutable;
use Drupal\views\Plugin\views\filter\StringFilter;
use Drupal\views\Plugin\views\filter\ManyToOne;
use Drupal\views\Plugin\views\query\Sql;
use Drupal\views\Views;
use Drupal\views\Plugin\views\relationship\RelationshipPluginBase;
use Drupal\Views\Plugin\views\join\JoinPluginBase;
use Drupal\views\Plugin\ViewsHandlerManager;
use Drupal\search_api\Plugin\views\query\SearchApiQuery;
/**
* Filters by Node Group.
*
* @ingroup views_filter_handlers
*
* @ViewsFilter("node_group_filter")
*/
class NodeGroupFilter extends SearchApiFilter {
/**
* The current display.
*
* @var string
* The current display of the view.
*/
protected $currentDisplay;
/**
* {@inheritdoc}
*/
public function init(ViewExecutable $view, DisplayPluginBase $display, array &$options = NULL) {
parent::init($view, $display, $options);
$this->valueTitle = t('Search API Node Groups');
$this->definition['options callback'] = [$this, 'generateOptions'];
$this->currentDisplay = $view->current_display;
}
/**
* Helper function that generates the options.
*
* @return array
* An array of group labels.
*/
public function generateOptions() {
// Assuming sbn_get_group_labels() returns an associative array of group labels
return sbn_get_group_labels();
}
/**
* {@inheritdoc}
*/
public function query() {
if (!empty($this->value) && is_array($this->value)) {
$this->getQuery()->addCondition('sbn_add_nodegroup_str', $this->value, 'IN');
}
}
}
But, views doesn't even recognize it. I can't find it in "Add filter criteria" to select it Can you see what the problem is?
Thanks!
@druken monkey
Thank you so much! Worked like a charm. For anyone else who runs into this situation:
First get the label, machine name and path property for the field for which you want a views filter. These can be found in Search API datasource fields. Also get the ID of the index.
In hook_views_data_alter:
////////////////////////////
// node group filter plugin
////////////////////////////
// Targeting the specific Search API index and field.
if (isset($data['search_api_index_default_solr_index']['label_2'])) {
// Change the filter plugin to 'search_api_options'.
$data['search_api_index_default_solr_index']['label_2']['filter']['id'] = 'search_api_options';
// Set an options callback to your custom function.
$data['search_api_index_default_solr_index']['label_2']['filter']['options callback'] = 'sbn_get_group_titles';
}
The only other thing you'll need is the function to get the group titles (if you don't already have one)
/**
* Options callback to provide the list of group titles.
*/
function sbn_get_group_titles() {
$options = [];
// Query to load all group entities or use an appropriate method to fetch group titles.
$groups = \Drupal::entityTypeManager()->getStorage('group')->loadMultiple();
foreach ($groups as $group) {
$options[$group->id()] = $group->label();
}
return $options;
}
That's it. You should be good to go. Just add the filter to your view:
p.s.
This is the general structural guidance I tried to follow to accomplish the custom views search api solr filter:
- https://www.drupal.org/project/search_api/issues/3003129 💬 how to make a custom exposed filter for SOLR view with hook_views_data_alter Fixed
SomebodySysop → created an issue.
SomebodySysop → created an issue.
I honestly thought this was the way to do it:
<?php
namespace Drupal\sbn\Plugin\views\filter;
use Drupal\search_api\Plugin\views\filter\SearchApiFilter;
use Drupal\views\Plugin\views\display\DisplayPluginBase;
use Drupal\views\ViewExecutable;
/**
* Filters by Node Group.
*
* @ingroup views_filter_handlers
*
* @ViewsFilter("search_api_node_group")
*/
class SearchApiNodeGroupFilter extends SearchApiFilter {
/**
* {@inheritdoc}
*/
public function init(ViewExecutable $view, DisplayPluginBase $display, array &$options = NULL) {
parent::init($view, $display, $options);
$this->valueTitle = t('Search API Node Groups');
$this->definition['options callback'] = [$this, 'generateOptions'];
}
/**
* Helper function that generates the options.
*
* @return array
* An array of group labels.
*/
public function generateOptions() {
// Assuming sbn_get_group_labels() returns an associative array of group labels
return sbn_get_group_labels();
}
/**
* {@inheritdoc}
*/
public function query() {
if (!empty($this->value) && is_array($this->value)) {
// Retrieve the underlying Solr query.
$solr_query = $this->query->getSolrQuery();
// Construct the filter query.
$filter_query = 'sm_sbn_add_nodegroup_str:(';
$filter_queries = [];
foreach ($this->value as $val) {
// Escape special characters in Solr query syntax.
$escaped_val = $solr_query->getHelper()->escapePhrase($val);
$filter_queries[] = '"' . $escaped_val . '"';
}
$filter_query .= implode(' OR ', $filter_queries) . ')';
// Add the filter query to the Solr query.
$solr_query->addFilterQuery(['key' => 'sm_sbn_add_nodegroup_str', 'query' => $filter_query]); }
}
}
But Views won't even recognize it as a filter. I know sm_sbn_add_nodegroup_str is the correct solrid, so I don't know what else to try. Open to suggestions!
Went through today and tried to duplicate the issue. Attached are results: https://www.drupal.org/files/issues/2023-12-05/2023-12-05%20activity%20l... →
Short of it, couldn't duplicate. The only significant difference I could glean is that on the 3rd, the theme was changed (upgraded/modified).
I also did a search for Contrib and Core modules that have generateSampleValue:
Contrib modules
[root@ip-172-31-13-94 modules]# grep -R generateSampleValue *
contrib/redirect/src/Plugin/Field/FieldType/RedirectSourceItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
contrib/devel/devel_generate/README.md:implement `\Drupal\Core\Field\FieldItemInterface::generateSampleValue()`.
contrib/devel/devel_generate/README.md:see: https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Field!FieldItemI…
contrib/entity_reference_revisions/src/Plugin/Field/FieldType/EntityReferenceRevisionsItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
If I remove the tests from Core results:
[root@ip-172-31-13-94 modules]# grep -R generateSampleValue *
comment/src/Plugin/Field/FieldType/CommentItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
datetime/src/Plugin/Field/FieldType/DateTimeItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
datetime_range/src/Plugin/Field/FieldType/DateRangeItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
file/src/Plugin/Field/FieldType/FileItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
file/src/Plugin/migrate/destination/EntityFile.php: $value = UriItem::generateSampleValue($field_definitions['uri']);
file/src/Plugin/migrate/destination/EntityFile.php: // generateSampleValue() wraps the value in an array.
file/tests/src/Kernel/FileItemTest.php: // Test the generateSampleValue() method.
image/src/Plugin/Field/FieldType/ImageItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
layout_builder/src/Plugin/Field/FieldType/LayoutSectionItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
link/src/Plugin/Field/FieldType/LinkItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
migrate/src/Plugin/migrate/destination/EntityContentBase.php: $values = $field_type_class::generateSampleValue($field_definition);
options/src/Plugin/Field/FieldType/ListItemBase.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
path/src/Plugin/Field/FieldType/PathItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
text/src/Plugin/Field/FieldType/TextItemBase.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
user/src/Plugin/migrate/destination/EntityUser.php: $name = UserNameItem::generateSampleValue($field_definitions['name']);
user/src/Plugin/migrate/destination/EntityUser.php: $mail = EmailItem::generateSampleValue($field_definitions['mail']);
user/src/StatusItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
user/src/TimeZoneItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
user/src/UserNameItem.php: public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
[root@ip-172-31-13-94 modules]#
The site worked on is not open to the public yet, so the only activity that day and today was the essentially layout stuff.
Can anyone think of any circumstance where 50,000 sample files would be generated within the space of a few hours?
The person is going to work on the site in the am. I will ask him to note the number of files in the Solr index before he starts and after, and what modules he uses.
SomebodySysop → created an issue.
@msnassar The patch in #12 works for all versions of Drupal (9/10) or all versions of Group? We are running Group 2.x.
@John Franklin, Thank you! Again!
To be clear, this is how it worked in Drupal 7/8/9
https://www.drupal.org/files/issues/2023-11-24/demonsration%20of%20how%2... →
Any css to be able to indent the lists themselves?
Thank you! That solves the indent problem, but still unable to indent the lists themselves.
Uploaded this video to demonstrate the issue: https://www.drupal.org/files/issues/2023-11-24/demonstration%20of%20how%... →
Attempts to indent the list only indents the list text. The bullet points stay aligned to left border.
SomebodySysop → created an issue.
What about reflecting the non ol|ul indents?
In my examples, I am not using li tags but straight indents, which are in the editor but not reflected in the theme?
To be clear, the indents do appear in the editor (ckeditor5 - Drupal 10.1.6)
But, not the theme:
The issue is supposed to be resolved in 4.1x branch, but I'm running 4.1.0-beta1 and getting the same errors. Same thing with the -dev version.
https://www.drupal.org/project/auditfiles/issues/3400341 🐛 Crash errors on "managed not used" and "used not managed" links Active
Help!
Also found I am getting this warning:
Warning: Undefined variable $rows in Drupal\auditfiles\Form\AuditFilesManagedNotUsedForm->buildListForm() (line 142 of modules/contrib/auditfiles/src/Form/AuditFilesManagedNotUsedForm.php).
Drupal\auditfiles\Form\AuditFilesManagedNotUsedForm->buildListForm(Array, Object) (Line: 21)
Drupal\auditfiles\Form\AuditFilesManagedNotUsedForm->buildForm(Array, Object)
call_user_func_array(Array, Array) (Line: 536)
Drupal\Core\Form\FormBuilder->retrieveForm('audit_files_managed_not_used', Object) (Line: 283)
Drupal\Core\Form\FormBuilder->buildForm(Object, Object) (Line: 73)
Drupal\Core\Controller\FormController->getContentResult(Object, Object)
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 592)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 181)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 19)
Drupal\sbn\Middleware\Redirect->handle(Object, 1, 1) (Line: 58)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 42)
Drupal\tracer\StackMiddleware\TracesMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object, 1, 1) (Line: 704)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)
I am getting the exact error on a new Drupal 10 install. Well, it's actually a 9.5.11 site that was upgraded to 10.1.16 and AuditFiles installed after the upgrade.
But, I'm only getting it with "usednotmanaged".
Any guidance on how to fix?
Drupal\Core\Entity\Exception\UndefinedLinkTemplateException: No link template 'canonical' found for the 'paragraph' entity type in Drupal\Core\Entity\EntityBase->toUrl() (line 196 of core/lib/Drupal/Core/Entity/EntityBase.php).
Drupal\Core\Entity\EntityBase->toLink() (Line: 130)
Drupal\auditfiles\Form\AuditFilesUsedNotManagedForm->Drupal\auditfiles\Form\{closure}()
array_reduce() (Line: 140)
Drupal\auditfiles\Form\AuditFilesUsedNotManagedForm->buildListForm() (Line: 21)
Drupal\auditfiles\Form\AuditFilesUsedNotManagedForm->buildForm()
call_user_func_array() (Line: 536)
Drupal\Core\Form\FormBuilder->retrieveForm() (Line: 283)
Drupal\Core\Form\FormBuilder->buildForm() (Line: 73)
Drupal\Core\Controller\FormController->getContentResult()
call_user_func_array() (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 592)
Drupal\Core\Render\Renderer->executeInRenderContext() (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext() (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 181)
Symfony\Component\HttpKernel\HttpKernel->handleRaw() (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle() (Line: 19)
Drupal\sbn\Middleware\Redirect->handle() (Line: 58)
Drupal\Core\StackMiddleware\Session->handle() (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle() (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass() (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle() (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle() (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle() (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle() (Line: 704)
Drupal\Core\DrupalKernel->handle() (Line: 19)
Installed the latest dev version of highlight in my 10.1.5 site.
https://www.drupal.org/project/highlight/releases/8.x-1.x-dev →
as per: https://www.drupal.org/project/highlight/issues/3287892#comment-15312879 📌 Automated Drupal 10 compatibility fixes Fixed
But I get this error every time I go into "Structure" or "Enable"
Warning: Undefined array key "url" in Drupal\Core\Asset\JsCollectionOptimizerLazy->optimizeGroup() (line 174 of core/lib/Drupal/Core/Asset/JsCollectionOptimizerLazy.php).
Drupal\Core\Asset\JsCollectionOptimizerLazy->optimizeGroup(Array) (Line: 183)
Drupal\system\Controller\AssetControllerBase->deliver(Object, 'js_q-CstTboDpeq_U8vRaN4F4gs80cO_uVUdnLZXGfqSVA.js')
call_user_func_array(Array, Array) (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 592)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 181)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 19)
Drupal\sbn\Middleware\Redirect->handle(Object, 1, 1) (Line: 58)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 42)
Drupal\tracer\StackMiddleware\TracesMiddleware->handle(Object, 1, 1) (Line: 34)
Drupal\webprofiler\StackMiddleware\WebprofilerMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object, 1, 1) (Line: 704)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)
Warning goes away when I uninstall Highlight.
I got this error when trying to install Search API 8.x-1.x-dev in Drupal 9.5.11:
Problem 1
- drupal/search_api[dev-1.x, 1.x-dev] require drupal/core ^10.0 -> found drupal/core[10.0.0-beta1, ..., 10.1.6] but the package is fixed to 9.5.11 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.
- drupal/search_api 1.x-dev is an alias of drupal/search_api dev-1.x and thus requires it to be installed too.
- Root composer.json requires drupal/search_api 1.x-dev@dev -> satisfiable by drupal/search_api[1.x-dev (alias of dev-1.x)].Use the option --with-all-dependencies (-W) to allow upgrades, downgrades and removals for packages currently locked to specific versions.
@John Franklin, Thank you!
SomebodySysop → created an issue.
SomebodySysop → created an issue.
@superman369
I actually forgot to do this before I entered the composer require command:
composer config --merge --json extra.drupal-lenient.allowed-list '["drupal/highlight"]'
So I ended up having to remove highlight (and vefl) because the patched versions would not upgrade to Drupal 10.
However, after upgrading the site to Drupal 10.1.5, I did go back and do this:
Entered patch info into composer.json:
"drupal/vefl": {
"Drupal 10 support": "https://www.drupal.org/files/issues/2023-03-17/vefl-drupal10-compatibility.patch"
},
"drupal/highlight": {
"Automated Drupal 10 compatibility fixes": "https://www.drupal.org/files/issues/2022-06-15/highlight.1.x-dev.rector.patch"
}
Entered these on the command line:
composer config --merge --json extra.drupal-lenient.allowed-list '["drupal/vefl"]'
composer config --merge --json extra.drupal-lenient.allowed-list '["drupal/highlight"]'
Then executed:
composer require 'drupal/vefl:^3.0' 'drupal/highlight:1.x-dev@dev' -W
And, darned if they didn't install!
Thank you!
Thank you! I've got 8 more sites to go, so this will be extremely helpful.
So, this is not a bug, per se, but a problem, I assume, with the size of my index.
This was not actually the entire solution, but was part of it:
https://www.drupal.org/forum/support/post-installation/2018-10-03/attemp... →
Exact steps to resolve:
Increase timeout in /etc/php.ini and httpd.conf
Restart web server
drush sqlq "DELETE FROM semaphore WHERE name = 'cron';" --uri=master1and1-9.schoolboard.net
Gets rid of the cron semaphores in database
https://www.drupal.org/forum/support/post-installation/2018-10-03/attem → …
drush cr --uri=master1and1-9.schoolboard.net
Clears cache so that next cron run will read most current /etc/php.ini maximum execution time
Reduce Index batch size.
The default is 300 but I reduced it to 5.
Re-Index
Had to repeat the above a few times, but eventually it got through the indexing.
So, this is not a bug, per se, but a problem, I assume, with the size of my index.
This was not actually the entire solution, but was part of it:
https://www.drupal.org/forum/support/post-installation/2018-10-03/attemp... →
Exact steps to resolve:
Increase timeout in /etc/php.ini and httpd.conf
Restart web server
drush sqlq "DELETE FROM semaphore WHERE name = 'cron';" --uri=master1and1-9.schoolboard.net
Gets rid of the cron semaphores in database
https://www.drupal.org/forum/support/post-installation/2018-10-03/attem → …
drush cr --uri=master1and1-9.schoolboard.net
Clears cache so that next cron run will read most current /etc/php.ini maximum execution time
Reduce Index batch size.
The default is 300 but I reduced it to 5.
Re-Index
Had to repeat the above a few times, but eventually it got through the indexing.
Here is where it gets confusing.
In my dblog I get:
type: search_api
location: search_api_attachments
message: Could not load the following items on index Solr sbn01 index: "entity:file/28364:en".
I get the message for file entities 28363 through 28369.
The problem is, my file fids end at 28362. There are no fids in my file_usage or file_managed tables with these fids. I have repeatedly cleared the queue before re-indexing, but keep getting the same timeouts.
I don't know if this is an issue with search_api or search_api_attachments.
In any event, I can't get anywhere because the htmlfilter keeps crashing the site.
SomebodySysop → created an issue.
SomebodySysop → created an issue.
Did you convert the text formats to CKEditor 5 before upgrading? It seems not.
Yes. That was exactly the problem. Lesson learned.
Had to use drush to delete the corrupted text formats. Used the UI to create the new text formats using CKEditor5.
In the future, always replace the ckeditor4 text formats with ckeditor5 before upgrading to Drupal 10.
@superman369
Thanks a lot. This is very good instruction. Unfortunately, I've run into a Symfony compatibility issue (because of my PHP v. 8.1.14) that is holding things up. Once I get that resolved, I'll post my progress back here. But, it looks like it's going to work.
@superman369
Thank you for the reply. In order to execute the dry run, I need to update my composer.json for D10 upgrade. In reading the Lenient read.me, it seems like it is designed to work with installs rather than upgrades.
So, I will try uninstalling the modules, see if I can upgrade, then try re-installing with Lenient and see what happens.
Did you uninstall, upgrade to D10, then re-install the patched modules? Or, did Lenient allow you to do the D9 - D10 upgrade with patched modules in place?
@maxilein
Thanks for the response. I don't understand what you mean when you say:
If vefl is enabled and patched: Just delete all D9 files (keep contrib and custom whatever)
Copy D10 manually to the right places. Start up. Should work.
Do you mean?
Uninstall (and remove) all incompatible D9 modules, then upgrade to D10, then re-install the D9 modules patched to work in D10?
Or, are you talking about manually downloading all the files and folders that make up D10 and copying them to my D9 install? If this is what you mean, what about composer.json file? How do I sync composer without corrupting it?
Thanks for the assistance!
in PATCHES.txt
This file was automatically generated by Composer Patches (https://github.com/cweagans/composer-patches)
Patches applied to this directory:Drupal 10 support
Source: https://www.drupal.org/files/issues/2023-03-17/vefl-drupal10-compatibili... →
In vefl.info.yml
name: Views Exposed Form Layout
description: Provides functionality to output Views exposed filters in layout.
package: Views
type: module
core: 8.x
core_version_requirement: ^8 || ^9 || ^10dependencies:
- drupal:layout_discovery
- drupal:views# Information added by Drupal.org packaging script on 2020-02-08
version: '8.x-3.0'
project: 'vefl'
datestamp: 1581161125
Did this again:
composer config --merge --json extra.drupal-lenient.allowed-list '["drupal/vefl"]'
drush cr
composer prohibits drupal/core 10.1.5
And sure enough, this is still there:
drupal/vefl 3.0.0 requires drupal/core (^8 || ^9)
I looked at composer.json, and I see this:
"drupal-lenient": {
"allowed-list": ["drupal/vefl", "drupal/vefl"]
}
},
So, I'm just not sure what step I am missing.
Thank you for the reply. So, what exactly does this do?:
composer config minimum-stability dev
composer require mglaman/composer-drupal-lenient
composer config --merge --json extra.drupal-lenient.allowed-list '["drupal/vefl"]'
Because, when I run it, then run composer prohibits drupal/core 10.1.5, I see see drupal/vefl listed.
Perhaps I am not using it correctly?
If someone has done this, how did you get past this limitation?
I've applied the patch, but composer prohibits still won't let me upgrade to Drupal 10:
composer prohibits drupal/core 10.1.5
drupal/core-recommended 9.5.11 requires drupal/core (9.5.11)
drupal/highlight 1.0.0 requires drupal/core (^8.8 || ^9)
drupal/vefl 3.0.0 requires drupal/core (^8 || ^9)
Even though this is what is in the highlight.info.yml file:
name: Highlight
description: Highlight key words from search results.
package: Search
core_version_requirement: ^8.8 || ^9 || ^10
type: module
configure: highlight.settings
# Information added by Drupal.org packaging script on 2021-05-20
version: '8.x-1.0'
project: 'highlight'
datestamp: 1621541825
What can I do to get past this?
I've applied the patch, but composer prohibits still won't let me upgrade to Drupal 10:
composer prohibits drupal/core 10.1.5
drupal/core-recommended 9.5.11 requires drupal/core (9.5.11)
drupal/highlight 1.0.0 requires drupal/core (^8.8 || ^9)
drupal/vefl 3.0.0 requires drupal/core (^8 || ^9)
Even though this is what is in the vefl.info.yml file:
name: Views Exposed Form Layout
description: Provides functionality to output Views exposed filters in layout.
package: Views
type: module
core: 8.x
core_version_requirement: ^8 || ^9 || ^10
dependencies:
- drupal:layout_discovery
- drupal:views
# Information added by Drupal.org packaging script on 2020-02-08
version: '8.x-3.0'
project: 'vefl'
datestamp: 1581161125
What can I do to get past this?
I have a question about creating a new core.
This is my normal process:
from /var/solr/data:
mkdir newCore
mkdir newCorei/data
cp -fr oldCore/conf/ newCore
Then create the core in the Solr dashboard and "Add Core"
Sounds like the only thing different about this process will be to upload the configuration files from configs.zip to newCore/conf instead of copying them from oldCore.
I know this is a composer question, but I just finally got to Group 2.2 after nearly since the first effort. In reading about what it takes to go from 2.x to 3.x, I'm terrified.
Currently, my composer settings want to automatically upgrade me from Group 2.2.0 to 3.2.0. How can I change my composer settings to not allow any automatic upgrade to 3.x?
I know this probably isn't the place for this, but I expect everybody here knows how to do it.
OK, so everything seems to have worked out fairly well. I only ran into these two hiccups that apparently weren't in my original notes:
change from getContentByEntityId to getRelationshipsByEntity
addContent() method is not defined for the Group class.
//
https://www.drupal.org/node/3292844#:~:text=,on%2027%20June%202022 →
changed to:
$group->addRelationship($node, $pluginId);
Appreciate the assistance!
Just finished unit testing permissions and it appears we are good to go with Group 2.2.0. Woo-hoo!
SomebodySysop → created an issue.
So, I am running quick node clone 8.x-1.16. If I want to install Group 2.x (coming from 1.x), so if I apply patch #5 I'm good.
But, what about this? https://www.drupal.org/project/quick_node_clone/issues/3306677#comment-1... ✨ Intercompatibility with Group module new versions Needs work
Does this need to be applied also, and if so, how?
Thanks!
Yes, I did review that page -- several times! I just didn't want to repeat the same mistakes. Thanks much for taking a look.
I do very much appreciate the discussion, and those notes helped me to get this far. However, I think it's a little unfair to say I "abandoned" the issue. "Postponed" is a far better word. And I had good reason. I was never able to get Group 2.x working. And, on top of that, I had these issues all related to Group 2.x incompatibilities with other modules I depended on:
Quick node clone incompatibility:
https://www.drupal.org/project/quick_node_clone/issues/3306677
✨
Intercompatibility with Group module new versions
Needs work
Group moderation incompatibility:
https://www.drupal.org/project/gcontent_moderation/issues/3309542
💬
Is group content moderation module compatible with Groups 2.0.0-beta3?
Needs work
Sitewide admin permissions:
https://www.drupal.org/project/group/issues/3318776 →
With all the back and forth, I eventually trashed the site I was working with:
https://www.drupal.org/project/group/issues/3330286 →
As I was slammed with trying to get Drupal 7 sites migrated to Drupal 8, and the Flexible Permissions module on which Group 2.x depended was still in alpha, it just made sense to postpone the effort until things calmed down a bit.
In addition, I got involved in moving the OpenAI project along, which resulted in me developing my own AI module:
https://www.drupal.org/project/openai/issues/3337774#comment-14908993
✨
Generate string vectors using OpenAI embeddings
Fixed
https://youtu.be/qgKiXYP8UBs
So, now that some of the incompatibility issues with other modules have at least been addressed, I turned my attention back to getting this working. Before I go and repeat the same mistakes as before, I was just hoping that someone who has already done this could take a look and let me know if it looks correct.
Given everything, I don't think that's an unreasonable request.
With respect to this specific change (which I am most worried about as it is not simply a renaming issue):
$group = Group::load($gid);
/** @var \Drupal\group\Plugin\GroupContentEnablerInterface $plugin */
$plugin = $group->getGroupType()->getContentPlugin('group_node:'.$node->bundle());
$group_content = GroupContent::create([
'type' => $plugin->getContentTypeConfigId(),
'gid' => $group->id(),
'entity_id' => $node->id(),
]);
$group_content->save();
Changed to:
// Updated code for Group 2.x:
$plugin_id = 'group_node:'.$node->bundle();
$relationship_type_storage = $this->entityTypeManager->getStorage('group_content_type');
$group_type_id = $group->getGroupType()->id();
/** @var \Drupal\group\Plugin\GroupRelationshipEnablerInterface $plugin */
# Group 2.x
$plugin = $group->getGroupType()->getPlugin($plugin_id);
$group_content = GroupRelationship::create([
'type' => $relationship_type_storage->getRelationshipTypeId($group_type_id, $plugin_id),
'gid' => $group->id(),
'entity_id' => $node->id(),
]);
$group_content->save();
These are the comments from Claude on the above change:
After reviewing your code changes again along with the Group 2.x upgrade documentation, I believe you have made the appropriate modifications to migrate from Group 1.x to 2.x. Here's a recap of the key changes:
- Use `getPlugin()` instead of `getContentPlugin()` to get the relationship plugin
- Load the relationship type storage directly instead of via the plugin
- Get the relationship type ID from the storage instead of the plugin
- Create the relationship using `GroupRelationship` instead of `GroupContent`
- Save the relationship correctly
These align with the recommended changes of:
- Using new 2.x methods like `getPlugin()` instead of old 1.x methods
- Accessing storages directly instead of via old APIs
- Getting IDs from storages instead of old plugins
- Creating/loading `GroupRelationship` instead of `GroupContent`
So in summary, yes your code modifications follow the prescribed upgrades for migrating custom code to Group 2.x based on the documentation. The main aspects like using new methods, accessing storages directly, and leveraging the GroupRelationship entity are addressed.
Barring any edge cases I'm not aware of, I would say your code changes are correct and should properly upgrade the integration to Group 2.x according to the provided upgrade guides and documentation.
SomebodySysop → created an issue.
Thank you!
SomebodySysop → created an issue.
Did you record it? If so, can you share the recording?
SomebodySysop → created an issue.
Finally figured it out. Not having any assistance other than ChatGPT, I was given this code to check for presence of job type
Bad
// Check if the "solrai_advanced_queue" job type exists.
$job_type = \Drupal::entityTypeManager()->getStorage('advancedqueue_job_type')->load('solrai_advanced_queue');
if (!$job_type) {
// If the "solrai_advanced_queue" job type does not exist, return an error message.
return "ERROR: solrai_advanced_queue job type does not exist!";
}
Through trial and error, finally lucked up on code that worked.
Good
$job_type_manager = \Drupal::service('plugin.manager.advancedqueue_job_type');
if (!$job_type_manager->hasDefinition('solrai_advanced_queue')) {
// The job type does not exist
return "ERROR: solrai_advanced_queue job type does not exist!";
}
Again, if there is documentation on this, please give me the link.
SomebodySysop → created an issue.
I actually figured out some stuff using ChatGPT.
However, I'd like to know how to create a queue programmatically. I created one in the UI. I've been trying to determine from Query.php how to do the same programmatically, but no luck so far.
This is what I have:
// Get the entity type manager service.
$entity_type_manager = \Drupal::entityTypeManager();
// Create the queue.
$queue = $entity_type_manager->getStorage('advancedqueue_queue')->create([
'id' => 'batchembed',
'label' => 'BatchEmbed',
'backend' => 'database',
'backend_configuration' => [],
'lease_time' => 300, // 300 seconds
'processor' => 'cron',
'processing_time' => 2700, // 2700 seconds
'remove_finished_jobs' => ADVANCEDQUEUE_REMOVE_FINISHED_JOBS_DAYS,
'remove_finished_jobs_value' => 7, // 7 days
'remove_finished_jobs_state' => ADVANCEDQUEUE_JOB_STATE_SUCCESS,
]);
// Save the queue.
$queue->save();
What am I missing?
SomebodySysop → created an issue.
6 months later, finally got a module working based on the prototype. Not ready for primetime, but it's working. The goal for me, and the object of this initial post, was to develop an embedding mechanism which integrates as seamlessly as possible with Drupal's existing content structure.
To that end, I have designed the SolrAI module to automatically sync Drupal content with it's embeddings. Right now, I sync the Solr index with it's Weaviate vector store objects, but the ultimate goal is to sync any content format with any vector store, including internal.
In case anyone else is working on something similar, this is how I did it:
Actually, I was able to accomplish my goals using the entity hooks. I wanted to use event subscriber instead, but just couldn't figure out the correct events to use. At any rate, for what I wanted to do (sync Solr index with vector store), I didn't need to monitor Search API (or Search API Solr) events. As Search API Solr automatically updates the Solr index on CRUD events, I only needed to monitor the Drupal entities for changes.
Again, for anyone else who bumps up against something like this, here is how I handled it:
No, I am only trying to track when a solr index object is affected: created, updated, indexed or deleted. Period.
Thanks for the info. I'll see what I can do.
Thank you. It looks like this is what I could use to react to any event on any Solr index object. Correct?
/**
* The postExecute event is thrown just after all execution is done.
*
* The event listener receives a Query instance and a Result instance.
*
* @var string
*/
public const POST_EXECUTE = PostExecute::class;
So,
use Drupal\search_api\Query\QueryInterface;
use Drupal\search_api\Query\ResultSetInterface;
use Drupal\search_api_solr\Event\PostExecute;
class MyModule {
/**
* The postExecute event is thrown just after all execution is done.
*
* The event listener receives a Query instance and a Result instance.
*
* @var string
*/
public const POST_EXECUTE = PostExecute::class;
/**
* Event subscriber callback for the postExecute event.
*
* @param \Drupal\search_api\Query\QueryInterface $query
* The query object.
* @param \Drupal\search_api\Query\ResultSetInterface $result_set
* The result set object.
*/
public function onPostExecute(QueryInterface $query, ResultSetInterface $result_set) {
// My code here.
}
}I look at the $query to find the function (update, create, delete, index) and the $result to get the item(s) affected.
On at least the right track here?
SomebodySysop → created an issue.
This module doesn't appear to be actively supported anymore. Is there an alternative for this type of functionality?
Does anybody have any documentation on how to migrate the D7 node gallery to any other photo gallery in D8/D9?