Leaf lifetime exceeds 60s in large migrations

Okay, let's fix this in Group then.

Actually, not sure about that. I've closed the other issue. Let me investigate if we can flag a group as "Created during this request" and run with that instead.

See the remark by @lind101 in https://www.drupal.org/project/subgroup/issues/3281672#comment-14876123 🐛 doAddLeaf & doRemoveLeaf: timestamp bug and scalability Fixed as well:

 protected function doAddLeaf(EntityInterface $parent, EntityInterface $child) {
    // This is a better implementation of the 60 second stuff
    /* @see \Drupal\subgroup\Entity\GroupSubgroupHandler::doAddLeaf() */
    if ($this->isLeaf($child)) {
      throw new InvalidLeafException('The provided entity is already a leaf, cannot add again.');
    }

Looks like he is saying that adding existing groups as a subgroup should just be possible. But you might be right that it creates all kinds of access issues.

Fuck it, let's just remove the assert. Right now you're not allowed to add existing groups via the UI and as long as you know what you're doing you could try to add them through code. E.g.: During a migration.

If you break your site because you wanted to add existing groups and did not take the necessary precautions, then that's on you. At least, until the module might one day support adding existing groups out of the box.

Rerolled #5 and committed.

I suppose you are implying that subgroup 1.0.x is EOL by not committing a patch for v1? Then it might be better to make a clear statement about this.

Uhm, good point. It supports Group v1 which is EOL, so I assumed Subgoup implicitly becomes EOL too. No point updating an extension to an EOL main codebase, I figured.

Is Group v1 really EOL? It doesn't look like:

Hidden inside an issue: https://www.drupal.org/project/group/issues/3313220#comment-14797177 🐛 permission_callbacks is not handled correctly RTBC

Well, that's a comment in an issue. But the status of the release is in the meta data of that release and that's still marked as support, although not recommended. But if the release will be marked unsupported (=EOL), then all sites that still use that release, will get red warnings in their sites that they are using an unsupported release. That can be done, but it will be a nightmare for all sites that still rely on it.

And that's still more than 90% of all Group module installations.

They may not be officially EOL, but both group and subgroup v1 do not get commits anymore. The same happened today for a major issue in this module: https://www.drupal.org/project/subgroup/issues/3281672#comment-15014560 🐛 doAddLeaf & doRemoveLeaf: timestamp bug and scalability Fixed
The patch for subgroup v1 was not committed.

It's not that they are officially EOL, but they are practically EOL. I simply do not have the resources to work on new stuff and maintain the old stuff now that we have an API (v2/v3) that makes maintenance and improvements so much easier. Which is why @dww was so kind enough to offer to help out with Group v1. But they, too, are sadly stretched for contrib time.

So I won't force people to quickly upgrade to v2, but it would be in everyone's best interest to do so given my limited resources.

The approach unfortunately reminds me of the story with Group 1.x RC5, after which the stable was released and all patches and RC5 defacto declared all instances/installations unusable.

Of course new stuff is great, new API, everything faster & easier. However, we currently have over 11k installs of Group 1, so this can't just be forgotten and send the module into EOL.

It takes a lot of resources, budget and developers to upgrade to Group 2/3, because usually for Group 1 a lot of customizations have been made or there are still additional modules in use that have Group 1 as a dependency but don't offer an update to v2/v3 yet.

An update from 1 to 2 is unfortunately also not done just like that, because we have no upgrade paths, many things have to be adapted. Here simply a little more time is needed until all people gradually upgrade the module.

Exactly so I see it critically that Group/Subgroup 1 module for Drupal 10 is no longer compatible in parts. So we run the risk that at the end of the year 11k installations can not upgrade to Drupal 10 and thus represent a security risk. This is a bit unbalanced in my eyes, we should collectively make Group/subgroup 1 ready for Drupal 10 as well so that everyone has enough time to deal with the Group 2/3 module upgrade.

Especially existing installations are important because there is already a lot of data there and that should not be disregarded. In a new installation is not bad because you take everything from scratch... that is unfortunately at the moment not quite optimal with EOL.

Thanks @zcht, I couldn't agree more. We have dozens of customer sites with over a thousand groups and subgroups each. This is still on Drupal 9 and we just haven't got a budget or time slot from the client yet, for when we could address the Group update to 2/3. We're completely lost here.

Couple of things I want to add here:

EOL

First and foremost, Group and Subgroup 1 are not EOL. They are simply not getting any new features, until they are EOL they will get security updates. Your site still works on v1 and it will continue to work on v1.

If Drupal ships with a new version that requires contrib to make changes, then it's not feasible for all of contrib to update all versions ever made since Drupal 8. You've got to pull the band-aid off at one point.

Having said that, I do acknowledge the majority of my users are still on v1 and it would be nice to support D10 to buy them some more time.

Resources

But then we get to my next point: Time and budget. My main priority is that Group keeps improving, growing and stays relevant and attractive for people to use in future projects. I allocate all of my time on that and I have been doing that free of charge for ten years.

I have made it abundantly clear that I do not have a fortune lying around allowing me to work on Group 24/7. I've brought it up at DrupalCons, on the module page, on Slack, in the issue queue, ... If people really want D10 support and given how small an effort it would be to get this working, I am surprised that out of 11.5k installs there is not one agency that can convince their client to fork out a little more budget to either sponsor their own developers to get this done or to hire me for a day or two to get this done.

Inconvenience

I am alone at this. If I need to make calls that inconvenience or upset part of my user base, rest assured that those were not made lightly. But given the above information, these tough calls have to be made or I risk getting stuck in support mode with no time left over for any new features or API improvements.

I wish I could do more, but I can't. Bad code got written, it needs to be cut and replaced. I too am human and therefore fallible, I too grow as a developer over the years and learn to write better code. It'd be a shame if I didn't.

I'm sorry if I can't keep supporting code from three years ago indefinitely as newer major Drupal versions come out. I'm sorry I had to cut support for D7 when the full version of v1 came out, but these are the harsh realities you face when you spend a decade writing code that hardly anyone seems to want to pay for.

I can count the amount of sponsors we've secured so far on one hand. Out of 13k installations, that's insane.

It's already late where I live, but one final thought:

If you buy a Samsung phone, you get Android updates for a few years. After that, your phone is no longer supported. It still works fine, you just don't get new and shiny stuff. This is a product you paid for and yet people seem to agree that after a while, Samsung no longer spends any time on it.

Maybe not the greatest analogy, yet it's a bit annoying to see large companies get away with scrapping support for paid items where contrib developers catch more heat if they want to drop support for free software. It should be the other way around.

Hi @kristiaanvandeneynde I can absolutely see where your frustration is coming from. Being a maintainer of many and sometimes also complex modules for the last 16 years, taught me those lessons as well. It can be really challenging, especially with major API changes in Drupal core. TBH, I wasn't aware of such major changes in Drupal 10, other than the deprecated functions. Therefore, I was under the impression, that the major changes in Groups 2/3 had been your free choice.

Looking at the situation from the "outside" it looks like it's pretty bad for all stakeholders. You've described your own situation perfectly and again, that's more than understandable. On the other hand, there is that fairly large user base on the old version, most of which most likely not aware of the situation and what's up for them at the horizon.

Maybe this needs some special attention from a team of people, and you/we may have to ask for help. Because my feeling tells me that just letting this go until everybody woke up, may lead to some disaster. If that could be prevented, let's pull all the strings required. I'm happy to help when I can, and I'm sure others will be there too. But hat should not be part of this issue, I guess.

@kristiaanvandeneynde I don't think anyone here wants to badmouth your work or criticize you personally.

Of course it is sometimes annoying when you invest a lot of time and have to please a lot of people. But I think Drupal has made the community great, the open source idea, everyone contributes something. Some more, some less, everyone who can contribute does it, be it patches, ideas for new features, bugfixes, UI/UX, documentation, YouTube tutorials, workshops, presentations, tutorials, 1-level support in issues, events or advertising for modules/Drupal, developer services.... everyone as he can. That's the great thing about OpenSource, but if you say I invested so much time here and I don't get money for it and I don't have time, then in my eyes it contradicts the OpenSource idea. I hope you can follow me.

Especially in the case of the Group 1 module: The appreciation of your work is the number of installations, even if it just means more work for you. But people really like what you made out of your idea and implementation, otherwise they wouldn't use this module to this extent.

I agree with you that there are changes in the API and thus new versions of Group are created, which also include other functions. The only thing to remember is the backwards compatibility to the most used version. It would not be a problem for many to switch to the next higher version if there was an update/upgrade path, as is the case with most modules. I don't think users say I only use group 1 and nothing else, everyone always wants to be up to date. But considering that going from Group 1 to 2 already has big hurdles, but then having to write a migration from Group 2 to 3 is not very purposeful and creates more problems. There is no documentation on how to do a migration, so learning by doing, then there are cases where you still have multisites and probably various other scenarios that I am not thinking of now.

I think it would be good if new features for Group 2/3 are released later, but before that an update path from Group 1 to 2/3 should be implemented. So people can update faster and you don't have to support the old version anymore, it will do itself and automatically.

Also that many patches in the Issues are not taken over in Group 1, but simply the Issues on the current Group 3 Dev with "needs work" are converted, is in my eyes also not a good handling.

As @jurgenhaas said, many probably don't even have the issue on their radar. When you HAVE to upgrade to Drupal 10, the big awakening/disillusionment comes. Not everyone is so active and deals with the problem, but if it turns out that you HAVE to take a security risk because you can't simply update the Group module, the resentment will probably be much greater.

As a suggestion, start a fundraiser like Matt Glaman did with phpstan for Drupal, I'm pretty sure many who use the module would be happy to donate something as well. But if you don't hear about it and only find out that it's not compatible because the maintainer doesn't have resources, that's very annoying for many. From version to version with the group module one has simply bellyache, because the modules are always incompatible among themselves. With a new installation this is not a problem, but already with existing data structures it is not so easy, because as I said you can't update just like that.

Another analogy that fits better in my opinion: Microsoft has always sold the Windows operating system, each new version cost money, so people didn't already have to pay for a new version every year and therefore used the version they were using until EOL or even longer and actively took the risk of using an insecure system. Apple did it differently, not only because of that Apple was able to establish itself better on the market: They included a new operating system as an update for free every year. This way, the systems not only remained secure and were provided with security updates, but new features were also introduced. EOL reached the update policy only after 10-13 years, although it would continue to work, but ultimately Apple also wants to sell new devices. That's how you make fans out of consumers... people are reassured that they get support and can use the system for a long time. MS is now doing the same thing after many years and is now trying to keep up.

Like I said, don't get me wrong, I totally understand you being frustrated. At the end of the day we are all in the same boat.

That's the great thing about OpenSource, but if you say I invested so much time here and I don't get money for it and I don't have time, then in my eyes it contradicts the OpenSource idea.

Sorry I vehemently disagree here. Open source means anyone can take my code, alter it to their will and I won't be able to stop them. You can even redistribute it and I won't be able to stop you either. It does not, however, mean that I am supposed to give you thousands of hours of my life for free. You have open source mistaken for slavery then.

It would not be a problem for many to switch to the next higher version if there was an update/upgrade path, as is the case with most modules.

But that's what I don't get. There is one. Coming from Group v1 you can easily go to v2 and just run some update hooks. The only thing that needs adjusting is custom code, contrib modules are (or should be) taken care of by their maintainers.

but then having to write a migration from Group 2 to 3 is not very purposeful and creates more problems

You don't have to in a rush. You probably have 2 years or more to get this done. If someone were to sponsor it, we could even write a boilerplate migration and a bunch of documentation on how to use it.

I also totally disagree with the description that @zcht gave of Open Source. It is perfectly normal to get paid for working on a open source project. And I do share @kristiaanvandeneynde ‘s astonishment that none of the agencies with dozens of customer sites relying on group and subgroup have stepped up to make sure that both modules will get D10 support. In my case it only is a single site and a single customer, so I cannot contribute much (besides my time).

Let us continue with this comment by @jurgenhaas:

Maybe this needs some special attention from a team of people, and you/we may have to ask for help. Because my feeling tells me that just letting this go until everybody woke up, may lead to some disaster. If that could be prevented, let's pull all the strings required. I'm happy to help when I can, and I'm sure others will be there too. But that should not be part of this issue, I guess.

Shall we open a new issue in the issue queue of the group module to attract attention for this problem? Would not that be the best way forward?

Or do we wait until @dww has got some spare time?

And I do share @kristiaanvandeneynde ‘s astonishment that none of the agencies with dozens of customer sites relying on group and subgroup have stepped up to make sure that both modules will get D10 support.

Well, our agency allocates 20% of all developer resources to d.o contributions. We can't spend cash on top of that.

And the second problem with that in the group module, there has never been a plan to make Group 1 D10 compatible. We got hit hard when we learned about the fact, that all our investment in the Group 1 solution is just gone.

Shall we open a new issue in the issue queue of the group module to attract attention for this problem?

Opening an issue might be part of what's needed from what I can see. But it will in no way attract attention from those who don't already participate in this discussion. What's needed is something like a task force that helps to resolve the forthcoming issues.

Well, our agency allocates 20% of all developer resources to d.o contributions. We can't spend cash on top of that.

Just to reiterate, patches are great contributions too. You don't necessarily need to pay me to do it in your stead.

And the second problem with that in the group module, there has never been a plan to make Group 1 D10 compatible. We got hit hard when we learned about the fact, that all our investment in the Group 1 solution is just gone.

It's more like I considered Group 1 "done" as soon as Group 2/3 were ready for beta/rc. I did not have any time left to work on v1 at that point and by the time v2/3 were done D10 was around and we ended up in the situation we are in now. v2/3 just took a really long time to develop.

Opening an issue might be part of what's needed from what I can see. But it will in no way attract attention from those who don't already participate in this discussion. What's needed is something like a task force that helps to resolve the forthcoming issues.

🐛 D10 compatibility Fixed was that issue and as you can see in the later comments, it's people helping out with patches and slicing them into easily reviewable pieces that made the issue move forward and got it committed. People even asked if I would consider committing backports and the answer was "yes". After that, all went quiet.

So please feel free to open a follow-up issue, linking to the original, so people can see how the other issue got committed. Then that could give some insight into how we might get things moving. You could also link to the discussion here.

Just to reiterate, patches are great contributions too. You don't necessarily need to pay me to do it in your stead.

Sure, that's what we always do. Or, if a patch is already present, we test and review that when possible.

And that's what happened here in this issue: I reported the problem, I participated in the discussion on how to resolve it, and somebody proposed a solution in a patch. Then it went quiet for 2 years and recently, the patch got committed to Group 2 and 3 but not to Group 1. As far as I'm concerned, that was the only reason that raised this discussion. Simply committing this existing and reviewed patch also to 1.x would be solution, just like you proposed in your previous comment.

Yeah the discussion got derailed to the whole D10 support thing, which implies more work. And someone was nice enough to spot this discussion and add Subgroup v1 to the spreadsheet for D10 compatibility: https://docs.google.com/spreadsheets/d/1jUv6jkLM9u49hqCcgf-4W9bIEKEhsfJ-...

As far as this issue is concerned, it's an easy backport to 1.x so I could add it. But the general idea is that some patches are really difficult to backport to 1.x due to highly different APIs and that's why my default mindset is now to not even bother any more as to not sow confusion why some patches get backported and some do not.

Essentially the main goal is: v1 keeps working but gets nothing new. Adding D10 support in a final release would be a nice compromise for "keeping things working" as far as I am concerned.

The same happened with https://www.drupal.org/project/subgroup/issues/3281672#comment-15005592 🐛 doAddLeaf & doRemoveLeaf: timestamp bug and scalability Fixed

There were patches for v1 and v2/v3 and only those for v2 and v3 were committed. The one for v1 is in production now at my site for some weeks and I was really surprised that a patch for a major issue did not get committed in the subgroup v1 branch.

@kristiaanvandeneynde: um, no, that's not what I meant by open source... of course nobody expects that someone develops something for free. There you have misunderstood me: with my explanation I said only that everyone participates in everything in the context of the personally possible, as I have already listed. Sorry, if it was misunderstood, but I wanted to add that at this point, so that there is no misunderstanding.

Essentially the main goal is: v1 keeps working but gets nothing new. Adding D10 support in a final release would be a nice compromise for "keeping things working" as far as I am concerned.

This is just such a perfect statement, I guess you would make the user base really happy by getting this established. And all the discussions, at least from within this issue that are related to the topic, will be resolved.

In my own words, keeping Groups 1 (and related modules) "minimally supported" for Drupal 9 and 10 (where 9 will be EOL by November), with only patches being back ported that fix bugs, no new or changed features, is what's needed. That takes the burden away from everyone, hopefully including yourselves and @dww

Automatically closed - issue fixed for 2 weeks with no activity.