🇬🇧United Kingdom @jonathanshaw

The only one of these edge cases that I actually care about in my project is the anonymous donor whose order can later be identified by having the order email match one of the users emails.

Given that I could workround this by updating the order entity retrospectively to add the customer, I'm sympathetic if you want to say "this contexts business is a nightmare, how about if I create the module without it and then we see if you'v got any budget left to worry about it?"

I'm struggling to understand the idea of "related contexts". I can see that we need to be flexible with context. Staff might detect duplicate donors and combine, or the website might even do it automatically (name/address similar). We might also need to split a donor apart if two individuals got accidentally merged. Or correct an error when a declaration was assigned to the wrong donor in case of two that are similar. However I see all of these as direct edits of the entities, rather than storing a separate entity to record that two contexts are related. I would like to have a goal of 1-to-1 correspondence Donor <=> individual. Could we then remove the idea of related contexts? It would mean that the list of declarations could be found by direct SQL query and we can use standard mechanisms like EVA to show the declarations for a Donor.

Imagine an organisation that has
1. CRM contact entities who may have offline ddeclararions recorded for them
2. Users able to access the website and make or cancel declarations for themselves
3. A commerce checkout where donors can mae a declaration at checkout

This creates a family of nasty edge cases:
A. Not all CRM contacts will have corresponding user entities.
B. A CRM contact may begin without a user but with an offline declaration, but later become linked up with a user and get a declaration or cancellation from that's users actions
C. A donation may be made with a declararion at checkout without being logged in. The user account could be created at the end of the checkout, or it could even never be set as the order customer but still identifiable as the same donor on the basis of email address.

The idea of 'contexts' was an atempt to handle these edge cases. The declaration stores the original context, there's no attempt to dynamically update what's stored on the declaration when further user/contact entitie are added/merged becaus that gets too complex and fragile. But nonetheless the system is able to pull these declarations together for a single donor when needed.

The only one of these edge cases that I actually care about in my project is the anonymous donor whose order can later be identified by having the order email match one of the users emails.

Store the address on the Declaration, correct at time of declaration. This is important if we need to move declarations between donors, or to unpick any errors in combining donors. The Donor address is used whenever we want the most recent known address. HMRC explicitly state that we don't need to update declarations if the address changes.

I've no problem with that as an implementation. I don't have clarity whether it's needed, but I've no problem with it and it feels plausible to me that it's a good solution.

Create an Entity type for Records such as cancellation or change of address. The alternative strategy of using revisions on the Donor is inherently inflexible because they are then locked into the specific Donor. NB I feel the Record entity approach is likely quicker to develop: the extra entity type is balanced by avoiding the need for Donor revisions and we get forms for free from the entity rather than having to design custom forms. In particular Cancellation was a bit strange as a Donor revision because it didn't change any Donor fields; it carried different information which probably we could only store by converting to a string and putting it in the log message.

The idea of a records entity type doesn't worry me as long as we also have a declarations entity type.

I think you probably still want donor revisions anyway as they're the most natural way of handling donor change of address.

I agree cancellation is strange as a donor revision. But there would be an evidence entity that would need linking to the donor as part of the cancelling which might hold a lot of the information.

I agree that a records entity type could well end up a nice clean solution. If you're going this way, I'd suggest not having a bundle entity type for the records. Instead maybe use bundle plugins from the contrib entity module. This means no config needed, and you get a class for each bundle where you can put specific logic, like putting cancellation logic in a postSave() on the cancel bundle.

Sure

I would use the flag so the web types don't appear in the list for staff to create (they only work as evidence for a self-declaration; even if staff create a commerce order on behalf of a donor they would need to use one of the other evidence types). Also the flag could affect access/validity: staff wouldn't need edit-access for 30 days to correct a data-entry error, hence we could allow claims to HMRC without the delay.

FWIW neither of those arguments seem right to me.

You don't need the flag *on the declaration entity* to remove web types in the list for staff. You need it on the evidence bundle. The staff form can show all the bundle types of evidence that have the bundle flag. The self-declare form doesn't overtly mention evidence bundles at all.

The validity argument is worth considering, but that seems like what we have the claimable_date field onthe declaration for. The self-declare form can just set the claimable date to today, the staff declaration recording form sets it to 30 days from today.

For example it seems to imply that Commerce-based declarations need to put the user as context rather than the order (else they will each have a separate donor referencing their separate order). If there is no user, perhaps we put the profile??

I think we better open an issue to explore this. It indicates that we have to be careful to make sure that we don't lose somehting of what the context system offered when we introduce donors.

#3 the evidence entity. I had been imagining a new specific new entity type rather than a dynamic reference to any entity. We can make bundles for each evidence type

Yes.

Probably next we need entity #4 the record that isn't a declaration - at least we need somewhere to store the key data e.g. for a cancellation: date of cancellation, and what does it cancel - a specific declaration (maybe relevant later for non-date-based declarations?) or all?

I'm doubtful we need a record entity type in addition to the evidence entity type. I think revision of the donor and declararion entity types, combined with the addition of new referenced evidence entities, is likely sufficient. This will probably mean things like cancellation need custom forms with a tailor made UX that do bespoke entity manipulation, rather than a simple use of an entity form. But that's ok.

However web and written declarations are fundamentally different evidence types for our software. The first is entered by the donor; has automatically generated evidence that references the order, and the displayed UI text; has corresponding code so is locked in the UI or at least can't be deleted or created. The latter is entered by staff, needs evidence such as a PDF, photo or email, and we should prompt staff to verify its various parts.

I'm not particularly fussed about 2 methods vs 3. But FWIW I don't yet fully share your perspective. Your argument there for me mostly points to the need to have 2 seperate forms, a self one and a staff one. I'm less certain whether that will involve an explicit need to flag web vs written or whether it will be obvious from the fact that declarant = creator.

Perhaps you mean the declared date, which I agree is important and we do have a field for.

Thankyou, you're right.

Create a multi-state validity, merging the two fields, good idea. This becomes very close to the status field, so I'm attracted to drop status as a separate computed field

Agreed. I suspect we need to postpone status until later when we have more clarity over when we need it and what for.

I feel inclined to avoid making the effective date be part of the validity state as that would require a cron hook, batching, and possible error cases

Possibly we will have a computed and a stored validiity (obviously not called the same!).

I will try inline entity form for creating the evidence entity.

I have UX concerns there as per #12.2 above. So by all means investigate, but please don't sink too much time into it. I have found IEF to be one of the most fragile, buggiest, complex and poorly maintained parts of the Drupal ecosystem. So I'm nervous on making this module's complicance critially dependent on significant customisations of it.

More specifically, AccessResult implements RefinableCacheableDependencyInterface so where we handle the query execution we could collect the cacheable metadata from the access result.

Thankyou, now I understand. The proposed queryAccess() handler method allows both modifying the passed in query, and returning an access result with cacheability metadata. Clever, although not a pattern I remember seeing in core before.

In principle having this on the handler seems good. But I see it as a small addition to the current system, not a simplification. And something that could be a separate issue.

I'm interested in the cacheability question you raise, but I don't understand how your handler proposal helps it. Can you say more?

Sorry to be slow to reply.

Replace "written_record_required" with "method" field with 3 states: oral; writing; web. The corresponding state of "written_record_required" is yes; prompt (staff to validate the evidence and tick a box); no.

I feel unsure here. My thinking on this hasn't fully adapted from the old plan where there where different bundles and so staff would encounter a page where they first selected the bundle with an explanation of when to use which, to the new plan where there may not even be declaration bundles and it's the evidence field referencing evidence entities of different type that handles the different delaration methods.

A few thoughts I'm having:

(1) The guidelines distinguish between oral and everything else. And we could actually treat web as a species or written. So maybe the method could just be a boolean is_oral.

(2) How do we create a good UX for staff to select which kind of evidence entity to attach? IEF for example tends to offer just a dropdown with the bundle label, which is not a reliable UX for an important compliance-relevant decision like this. Two possibilities:
(A) use a modal for selecting the evidence bundle when adding evidence. Maybe there's an IEF variant for this.
(B) Reverse the entity creation flow. First the staff choose the evidence bundle from a standard bundle-based entity add page, then they go to a declaration form prepopulated with empty evidence of the chosen bundle.

(3) If we use option B, then we could programatically set the is_oral field depending on the evidence type chosen (if we had an is_oral field on the evidence type, specified by the sitebuilder).

I see that the word record is predominant in 3.10.1 so it's inconsistent! Also A few times it uses "written confirmation". The word "record" is least distinctive, used in many other ways a total of 101 times whereas statement is only used in this context.

"written statement" is used in 3.8 to describe a process for retrospectively validating invalid declarations. "written record" is used in 3.10.1 for what needs to be sent to oral declarants. The two are functionally similar and ideally our work on 3.10.1 should made it easy to support 3.8 in the future, but 3.8 is out of scope for now and a rarer use case. So that's an argument for "written record". "Written confirmation" could be argued to be good because it covers both situations without being overidentifed with either.

DONE Change field "invalid" to "valid". It seems more straightforward, and implies an active choice to demonstrate validity. Optional

Validity is an interesting one. There's 2 main flows in the guidance:
1. An oral declaration is not "effective" if there's no recording until a "written record" is sent. And it its "cancelled" within 30 days of the written record being sent then it is treated as if it's "retrospectively" cancelled and treated as if it were never effective.
2. An declaration with evidentiary or data problems can be deemed "invalid" retrospectively. A charity can "validate" it by sending a "written statement". If undelivered, then treat as "cancelled". If not "cancelled" within 30 days, then the declaration can be considered "valid".

Notably, it's not obviously impossible to send multiple written statements if something weent wrong with the first one, or multiple evidentiary problems were found successively.

So there's 3 states:
1. Invalid/ineffective before written statement sent
2. Pending, will be effective in 30 days
3. No problems, all good

It seems like we might need:
1. An effective_date field that indicates the date from which the declaration can be relied on.
2. A mechanism to set the effective_date to 30 days subsequently when a written record/statement is sent.
3. A way to mark a declaration as cancelled from the moment it was created, being never effective.

Not sure how best to handle all this.

Sure, merge it.

SmartDateDurationFormatter already passes the timestamps.

As I uderstand it: SmartDateDailyRangeFormatter, SmartDateRecurrenceFormatter and SmartDateRecurTrait all seem to built of top of the smartdate field type, which stores timestamps not iso_8601 strings and so already passes timestamps to the augmenter. And as the files aren't opted in to strict types, we don't need to explcitly cast to integer.

So I think we are ok?

The reason why this wasn't done previously is probably this code:

      // If only one of start and end are displayed, alter accordingly.
      if ($parts['start'] xor $parts['end']) {
        if (in_array('start', $parts)) {
          $end_ts = $start_ts;
        }
        else {
          $start_ts = $end_ts;
        }
      }

The code modifies the timestamps permanently, effecting in all the subsequent contexts in which they are used.

Some of these contexts - like the augmenter - need the original stord timestamps, but some of them probably want these modified timestamps. In my fix here, I have passed the stored timestamps to the augmenter only. Figuring out which other contexts need the stored timestamps and which the modified timestamps is beyond me though, that needs deep understanding of this module.

Fortunately, we don't have to solve that here. What I've done in this MR is a step forward, even if there's more that could be done.

jonathanshaw → created an issue.

Does making it optional solve the security issue sufficiently for ✨ Allow users to login using either their username OR their e-mail address Active to advance? The IS needs to explain this aspect I think.

Move the charity and context fields into it (from declaration)

This is the bit I hesitate about.

I think perhaps it needs its own context field, but the declaration still needs a context field too. So the declaration finds its Donor indirectly via the context.

But then from a compliance point of view I guess your thinking its better to be more fixed about the donor's address "on" the declaration. That's a good argument.

But how do we ensure single donor reuse across multiple declarations? By finding any existing donor via context when creating the declaration. So context is still a crucial intermediary. Maybe this is OK.

The name can change and still be the same person.

This point makes me nervous, it would look really bad at audit to be offering a name and adress for a person who was not the one who made the declaration. I think maybe we need a checkbox or confirmation form where they can explicitly choose to apply the new default customer address to their gift aid record.

- Some need it conditionally, depending on 'Advised of liability' (and we can write clear advice to staff for filing out 'Advised of liability')

That's a good idea.

For my legacy paper forms scenario, I can create a bundle specific to that scenario where confirmations are always sent, not optional for staff to decide. It means I need a seperate bundle for new (non-legacy) paper forms, but that's acceptable solution for an edge case like this.

Actually staff shouldn't be creating declarations for the bundles that don't need validation (such as web self-declaration), and we could enforce that (with admin override).

Agreed. We could have permission per bundle, but actually I think it's slightly confusing to use the normal Drupal idiom for something that has compliance implications. Better probably to have a bundle setting "Allow declarations of this type to be recorded by other users." because that way we can add more explanatory notes etc in the UI. The permissions UI is not a good place for stuff that has complicance implications.

My current preferred idea is a hybrid of the 2 solutions: a cancellation would automatically update all declarations to alter their validity. We could separate fields for the original end date (immutable, entered in the UI) and the current end of validity (calculated, read/only in the UI).

I like it. It's very robust and clear. I had hoped to not need it, but it's good and not so hard really.

Note that this "end of validity" is just a "shortcut" which could be recalculated at any stage (someone could write code to do this automatically). This is great for untangling data inaccuracies, such as the "complication" in the IS or if one person has ended up with 2 distinct contexts (database records) which need to be combined.

My concern there is distinguishing between originally specified end dates and later applied ones. So maybe 2 different fields?

In 🌱 Product requirements Active I found this interesting line in HMRC guidance:

3.9.3 If a donor wishes to alter the description of the gift or gifts shown on a declaration [i.e. start and end dates] , they must cancel the declaration and make a fresh one.

I feel we should leave the site owner free to choose how they use declaration types. E.g. they might instead want declaration types to correspond to donation types: one for commerce, one for direct debit, one for a sponsorship form etc. If we have a field for "method" the software can calculate when a written record is required automatically (it would also be true if we are missing some information).

Evidence is required by HMRC - if we have a base field we can enforce that. We could make evidence a reference to a media type, and add a field to say what it is evidence for. Then we could write some code that checked that the current changes that are being saved have matching evidence.

The debate is good, but I stand by my position.

The key point is that what amounts to evidence, and what is a good UI for entering that evidence, is not something we can know. Only the charity will know its existing systems for capturing and storing evidence & how to integrate with them, and what the predominant use cases are for data entry in its situation and how best to optimise UX for them.

We can't know that a media reference field is the best UX for any or most use cases. For example, when we create an email bundle I will suggest that we put fields "email address" and "email text received" on it as that's the best way to create a gift aid trail for an email.

The bundles should focus on optimal UX for different declaration scenarios.

I'm imagining we should provide configured (so optional and alterable) bundles:
- commerce (i.e. checkout)
- web (i.e. self declare by web UI)
- email (staff declare)
- phone (staff declare)
- face to face (staff declare)
- letter (staff declare)
- paper form (staff declare)

The question of when to send a written confirmation is rather complex. I do agree it needs careful discussion. See 🌱 Product requirements Active 3.2.
It looks like site builders should be able to require a written confirmation on a bundle, disallow a written confirmation on a bundle, or make it optional as staff specify.

'Advised of liability' is an interesting one. HMRC seem quite hot on it as part of an audit trail. I have some concern that the fact that e.g. a checkout is configured to show the liability message could provoke a query - can we prove that on such and such a date the user actually saw a liability message and what exact message did they see? It makes me wonder whether it's good even for commerce, web, paper form contexts to have a field that we can set to indicate that the liability advice was given at time of declaration. I'm wondering if even if we prefill it and don't allow staff to untick it on some bundles, it might still be useful at audit and 'look more convincing'.

I have created 🌱 Product requirements Active as a replacement for this issue.

jonathanshaw → created an issue.

The profile type automatically comes with lots of free stuff, including a user tab, however there's a risk it will end up being not quite the stuff that we want and difficult/impossible to alter.

Yes.

So the choice between "Donor entity" and "gift aid profile type" is likely just a matter of ease of implementation.

Yes.

I feel that if we use the existing commerce profile, then we don't actually solve the Problem/Motivation in the IS: profile might get deleted too soon; order might not have an address; revisions are off by default; staff not required to provide evidence; lack of clarity which profiles relate to gift aid.

Good considerations.

I'm definitely open to the idea that it'd be no harder, and likely cleaner or more robust, to have a donor type.

One aspect of handling the commerce customer profile integration could be that if a user edits their default customer profile, then we add a checkbox saying "I have changed my name & tax address for UK Gift Aid too." and update the matching donor correspondingly. Or possibly, we should just assume this is what happens as long as the name is unchanged - HMRC would probably prefer our best guess as to the up-to-date contactable customer address, rather than what we can rigorously prove.

1 and 2 are what I was thinking, great.

I don't fully understand the first and second option in 3. I wonder about having a checkbox when staff create a new declaration with an end date "Enforce this end date on relevant previous declarations?"

we could make a Donor entity type

That's probably got advantages. My hesitation about using a solution other than profiles, is that if one is using commerce, then the customer naturally has a set of already provided addresses in the customer profiles, which it makes sense to allow as options.

My suspicion is that this will be hard to answer until we have a more precise articulation of the exact UI scenarios we need to cover.

Good spot.

Cache::invalidateTags([$old_host_entity->getRegistrationListCacheTag()]);

FWIW I would have done this in Registration::postSave().

jonathanshaw → created an issue.

In \Drupal\registration\RegistrationAccessControlHandler::checkAccess, there is no explicit check to grant access to users with the 'administer registration' permission.

No. But we do have:
$result = parent::checkAccess($entity, $operation, $account);

And EntityAccessControlHandler checks this permission:

protected function checkAccess(EntityInterface $entity, $operation, AccountInterface $account) {
...
    if ($admin_permission = $this->entityType
        ->getAdminPermission()) {
        return AccessResult::allowedIfHasPermission($account, $admin_permission);
    }
...
}

But we call that parent check AFTER we check for host/configured.

Yes, Queue::prepareJob() should be able to return NULL, meaning 'discard this job'. This makes sense because prepaeJob() relies on job type plugin's handleDuplicateJobs() method, which has the design feature that it can return nulls, meaning "discard the current because it's the duplicate of an already queued one".

Queue:

  protected function prepareJob(Job $job): ?Job {
    ...
      if ($duplicates = $this->getBackend()->getDuplicateJobs($job)) {
        $job = $job_type_plugin->handleDuplicateJobs($job, $duplicates, $this->getBackend());
      }
    }
    return $job;
  }

JobTypeInterface:

  /**
   * Handles existing jobs detected as duplicates when enqueing a new job.
   *
   * A function can be used to execute a range of different strategies with
   * regard to duplicate jobs:
...
   * - to discard the new job and leave the duplicate intact, return NULL.
...
   * @return \Drupal\advancedqueue\JobResult|null
   *   A new job to enqueue on the backend, or null if no new job should
   *   be enqueued.
   */
  public function handleDuplicateJobs(Job $job, array $duplicates, BackendInterface $backend): ?Job;

I think your concerns about audit are worth taking seriously for sure. Maybe revisions and Entity Reference Revisions for the profile reference field would address it. Or just store the address *at the time of declaration* on the declaration in an address field that doesn't get updated.

When you make a Gift Aid claim you don't show the declarations to the Inland Revenue or break it down by declaration. Instead the claim is specific to a financial year and shows the donors name, address and the total amount of donations in the year for that donor that you're claiming Gift Aid on. The address used should be the best address for the donor that you have, even if you're making a claim for 3 years ago when they gave you an older address.

So the concept of address is really more at the level of the donor rather than the declaration, even if it needs to be present on the declaration.

By cancelled, I mean to set the end date to now. I believe this solves the problem you refer to.

That sounds like my second solution from the IS?

I don't feel intuitively connected to your idea that there is a special active declaration, ... It seems that at best it could only apply ... at a certain moment in time... Mostly we will want to know if at least one declaration exists, i.e. hasActiveByAnyContext(). We might also need to put a link to one of the declarations, but it could potentially be chosen arbitrarily??

Good. Our thinking aligns. Ironically we each thought the other was saying the opposite!

create a new declaration when rights are granted, i.e. the date is moved later, and bulk update when rights are removed.

Yes!

In terms of retraction, then I could see that as a special case to handle by a user with admin permissions.

A good enough solution for now to be sure.

In the case of declarations moving into a context (a online donor with existing old declarations gets associated with a CRM contact who has recently cancelled gift aid)

I wonder if this module needs to take responsibility for that case??

Agreed. I think it's good to have some awareness of context edge cases in case we can sometimes avoid them. But let's not create elaborate mechanisms to handle them. Nice if we can document them.

I think we have the theoretical clarity we need, we can move on to implement.

As I pointed out before, this function is currently not used, so I feel it's premature to keep spending time altering it.

Agreed. Can you leave a todo flagging the question and otherwise leave the code in whatever state you think best?

john.oltman → credited jonathanshaw → .

I don't think it's particularly important, but thought the idea might be worth sharing.

jonathanshaw → created an issue.

See also ✨ Record donor name & address Active

Nice!

2) Sure I can wait if you like. However in my view, the current code is unacceptable in terms of audit requirements as declarations change their end date in baffling ways without an audit trail. We could safely remove it even if we don't know what we'll use instead.

For right now, I don't mind which you do. But the MR is weird in that it reintroduces $later, but also removes $newer. That will definitely produe bad results I would think. Originally bother were present.

(3) getActiveByAnyContext

If 2 declarations had the same end date, we should prefer the newer of the 2.

(5) The dates in Declaration entity.

I'd have had 3 concerns with the MR:
- it used \Datetime to get the current time, not Drupal's time service. That's untestable and not best practice.
- it seemed to remove the edge case handling associated with this comment:
// The declaration ends at the end of the day, not beginning of the day.
- I thought that anytime we stored a date in Drupal we needed to update the timezone to the storage timezone before we format it using the storage format. Because when it gets loaded it will be assumed that the stored string is in the storage timezone. Seems dangerous to break that assumption for any custom code accessing these fields.

I like your idea that maybe assuming Europe/London is a good rule of thumb in this module. And your point that the tax year end is in BST might be important.

My current thought is that getStartDate() and getEndDate() should rebuild the timezone to Europe/London:
$formatted = $stored->format('Y-m-d H:i:s');
$new = new DateTime($formatted, new DateTimeZone('Europe/London'));

Additionally, getEndDate() should use "23:59:59" as the time.

The donor has changed their mind. We can then mark the old declaration as cancelled

mark the old entry outdated: this is an updated preference

I don't think we can do either of these things. The problem is that declarations will often be partially overlapping: the old declaration may have a start date earlier than the newer one, and in that earlier period it may be the necessary declaration for donations made in that period; it's only superseded in the later period.

The basic questions with cancellation is "where do we store the new end date"? On the declaration we currently think is the active one, or on all of the relevant stored declarations? I'm thinking it's safer to apply it to all of them, in case some shenanigans changes which one is active.

Changes of start date should be very rare, and are hard to reason about; I'm happy to leave that to admins to figure out manually editing all the relevant declarations.

I feel it will help us to phrase things in terms of the standard Drupal terminology of CRUD.

I've actually been skittish about this. As you say, donors shouldn't be exposed to the complicated concept of declaration instances. But for things like cancellation, I'm not sure that non-admin staff should be either. We might need some simple forms that take the necessary entity updates, but which ones are conventional entity edit forms is not yet clear to me.

Creating a duplicate fails

I don't think so. It's a new source of legitimacy to a Gift Aid claim, even if Inland Revenue decided at audit that your previous declaration was invalid for some reason, you could still be covered if this new supposed duplicate with a different evidence trail was valid.

Update is allowed within a short time period to correct a clerical error and after that, there are access restrictions.

Yes, I've wondered about whether we would end up with a mechanism like that. Not yet fully clear to me. I guess there's always scope to upload additional evidence, so maybe the access control is at the field level.

Can cancel (set end date to 'now') but cannot set end date in the past. Maybe can also shorten or lengthen end date to a new future date

I'm thinking that it may be better on GiftAidOverviewContextController to show the user's current Gift Aid declaration status, and then have buttons like "Add declaration" or "Cancel" that go to specific forms with a UI and staff guidelines appropriate for that.

Delete is not allowed except for admin

Except within short time period as you suggest above.

jonathanshaw → created an issue.

Yes, if we use this for the access it makes sense to have it.

Feel free to reopen if you think that's right, I agree we definitely should have an issue for adding UI for customers to add/manage payment methods and it would be great to see your code for this.

(1) You're right, the removal is good.

(2) This is related to 🌱 The problem of changes Active . I'm OK with you removing the code here if necessary to pass tests, or leave the todo to resolve in the context of that issue.

(3) Consider this scenario. A user makes a declaration saying we can claim gift aid indefinitely. Later a user makes a declaration seperately saying we can claim gift aid this year. Does the later more limited one win out over the earlier one here? I was thinking it shouldn't because it might be an artifact of a more narrowly focused form or something, but now I doubt that and think maybe the newer one should be taken as a partial cancellation per 🌱 The problem of changes Active . So I suggest we need to revisit this question in that context, but the removal may well be OK. However, even if the removal is technically OK because the circumstances should not arise, I'm not sure whether or not it's good to use this point-of-loading to enforce the cancellation.

(4) The only thing that's making me hesitate about the fields is the commerce store, which has some of the same information. Probably that's just harmless duplication and we should put the fields in. Yes, fine to use address module.

5)

I fixed the start/end dates on Declaration, however on reflection I believe it's still not right.

I'm not sure what you mean. Maybe what you did in
https://git.drupalcode.org/project/gift_aid/-/merge_requests/3/diffs
I overlooked that earlier, there's various things about it that don't seem right to me at first glance.

I created ✨ Record donor name & address Active for address handling and added it to 🌱 [META] Roadmap Active .

I like the discussion on "advised of liability", oral and bundles a lot. I'm still not sure what's right, but I think it's important and sensible discussion. Worth its own issue.

jonathanshaw → created an issue.

Yeah, I figured this was a notepad of sorts. See 🌱 [META] Roadmap Active for priorities.

Add name and address required fields to the Declaration entity.

My thinking was that these belonged with the context, not per declaration. For example, on an order, the order's billing profile. Also, if a user changed their address, then that shouldn't have to be done on each declaration of that user. And if a user changed their address in the system for reasons unrelated to Gift Aid, then that should probably also be their new address for Gift Aid.

More generally though, we need to think about address complicance carefully:

If the charity is notified of a change in the donor’s name or address, it must keep a record of the updated information — this can be kept as an electronic copy such as a scanned document. The declaration itself does not need to be amended, but a record of the change should be kept on the charity’s database.

Add method (oral/written/self etc) and evidence (file attachments) fields to the Declaration entity

I was thinking to use bundles here, with different configured fields in different bundles. We should provide config for the common circumstances, but allow sites to modify further for their use case. As the fields aren't used programatically, it seemed better to allow configuration to do the work here?

and remove written_record_required from DeclarationType.
Add advised_of_liability field to the Declaration entity. If not advised, then written record is required.

I'm not sure. Oral seems to require a written record unless you have an audio recording of it? That's why I had this on the DeclarationType.

And AFAICS "advised of liability" is a requirement of any valid declaration at the point of declaration.

Add a page for a user to view/edit/create their own declarations and donations.

Yes, that's ✨ Fix GiftAidOverviewController Active although see 🌱 The problem of changes Active .

Enhance the declarations listings to cover everything needed for audit. Add some filters.

Nice.

Turn on revisions, perhaps require a revision log or generate automatically, write submission guidelines.

Yes, this is a very important area, linked to 🌱 The problem of changes Active .

Add a link from the order/donation to the declaration.

Yes. Or at least a Yes/no.

If possible add a link from the declaration to all the orders/donations. (3.28).

Out of scope for now.

Remove the restriction that declaration end date cannot be in the past (3.9.3).

This can be a form of cancellation with an existing declaration. See 🌱 The problem of changes Active .

Move the declaration form settings/building from GiftAidDeclaration to Charity to allow using the form elsewhere (e.g. the self-declaration page).

Code-reuse is good, but why Charity rather than a parent class I'm not sure.

Add support for tokens that refer to charity fields. Ensure that the charity name is included (3.9.2)

Twig us fields for free and is in many ways superior, so I'm happy to only support that.

jonathanshaw → created an issue.

jonathanshaw → created an issue.

I just noticed this charmer:

If a donor who’s made an oral declaration and been sent a written statement by the charity cancels their declaration within 30 days of being sent the written statement, the cancellation will have retrospective effect, so that it’ll be as if the declaration had never been made.

That retrospective effect needs careful consideration, I think it might be why I wanted written_record_sent_date despite it being technically redundant as you said. Because it would make it much easier to compute/display whether it was "safe" to claim a donation under a declaration.

Choose charity with drop-down rather than auto-complete
Shorten some very long field labels, instead putting information into the description (E.g. "Is this an enduring declaration that covers all donations within a period of time?")
Avoid using technical jargon such as entity in UI menu/ page title, e.g. "Declaration type entities" => "Declaration types"

All sound good, maybe a UI cleanups issue.

Put entity links on declaration listing

Very likely makes sense. Maybe we should have a listing issue, as you point out it should have more work to enhance auditability, but this is not the thing to tackle first.

Commerce pane improvements. Give information. "Thank you already covered". "Sorry not eligible". "£XX.XX of your order is eligible". Maybe these supersede the when options? Allow to configure the data parameters of the declaration?

Interesting. Would need more discussion in its own issue. I wonder if its necessary in the 95% use case.

Fix deprecation notices in test results

Great

In GiftAidDeclaration::submitPaneForm(), set the Declaration date to match order date

Maybe. I'm not sure quite when/if it matters. But it makes me think we should have an issue "Ensure compliance if an admin checkouts an order for a donor" as that scenario raises all kinds of hairy compliance questions.

Associate declaration with User rather than Order if available, then can drop second half of code in GiftAidRelatedContextsSubscriber. This should be much better performance and more intuitive.

We lose quite a lot of audit information about the context of the declaration. Maybe that doesn't matter. I have an attachment to the way we currently do it, but I think you are probably right.

Change some cases that are currently invalid declaration, into required fields?

Worth considering.

Possibly you want https://www.drupal.org/project/inline_entity_form_preview →

Nice, thanks Adam.

Shouldn't src/RegistrationValidationResult.php be implementing RefineableCacheableDependencyInterface not CacheableDependencyInterface?

Your change to the behavior of RegistrationValidationEventSubscriber::getCacheableMetadata looks like it break the behavior of registration_admin_overrides\EventSubscriber\RegistrationValidationEventSubscriber

I'm also not sure what that change does to the way the validator and constraint validators handle cacheability, but I imagine we have test coverage there.

Sorry to be slow in reviewing this. A few points:

1. In RegistrationType, shouldn't the workflow be able to affect cache context and expiry as well as tags? I'd be tempted to add the workflow as a cacheable dependency in the constructor.

2. In modules/registration_inline_entity_form/registration_inline_entity_form.module you didn't remove a now redundant addition of the entity as a CacheableDependency.

3. In HostEntity you've overridden getCacheMaxAge() in a way that stops us inheriting any lesser max age from a cacheable dependency.

4. The just-in-time updating we're doing in getCacheTags() feels odd. We have to repeat it across getCacheTags, getCacheContexts, and getCacheMaxAge. It's vulnerable to bugs like 2 above. We have to maintain logic that core will take care of for us. But I get your concern about settings not existing yet - I like what you're doing with registration_settings_list. How about if we add all the field, registration type and settings as cacheable dependencies in the HostEntity constructor, but then additionally add the registration_settings_list cache tag if the settings are unsaved? Should work and is neater.

5. addCacheableDependencies() feels like e legacy method that should maybe be deprecated. At the least, it's little confusingly named. But it does contain some interesting ideas ...

6. addCacheableDependencies() adds a registration_list cache tag. Which reminds me that isUserRegistrant(), and all the capacity methods, likely end up depending on registrations. Probably we should add that tag to the host_entity. If we don't, we need to add it to more validators. At the moment only HostHasRoomConstraintValidator has it.

7. addCacheableDependencies() adds session or user.permissions cache contexts. I've no idea why, but if they belong there then probably they belong on the host entity or some of the validators.

john.oltman → credited jonathanshaw → .

jonathanshaw → created an issue.

jonathanshaw → created an issue.

Charity has accessor methods for some fields that no longer exist so I removed them. The alternative of course would be to put the fields back.

New issue please. I'm not sure what's right here, that might get clearer. I suspect we need to display the charity number and address to users at some point.

Nice work! And challenging :)

I'm fine to leave commited the changes you've made, but some need further discussion in new more specific issues, ideally with the removed code in the IS for consideration, or a link to the commit in the MR here.

In GiftAidRelatedContextsSubscriber I removed the test against a draft order because for sure the order is draft during the checkout process when looking for an existing declaration.

New issue. Your point is valid, but I'm not sure a past declaration on a draft order should be considered to have been truly declared.

disabled the code in Declaration::postSave() for "If this declaration has had its end date edited, force the same end date on other ongoing declarations for the same context."

New issue please. Your points are good, but I had a good reason we should consider how to address.

I simplified the code in DeclarationStorage::getActiveByAnyContext() so it just returns the most recently declared, and removed the corresponding tests. It seems that it shouldn't really matter which one we return if multiple are relevant, and I couldn't conceive of any rule relating to start/end dates that actually works reliably.

New issue. That code looks ok to me.

The routes gift_aid.declaration_make and gift_aid.declaration_extend are apparently missing the corresponding form. I'm not sure if they are still needed - potentially we could reuse the add and edit forms??

Let's remove extend, I can't see an important case for extending a declaration rather than just making a new one.

For make, please open a new issue.

DeclarationFormTest has code that expects special button text changes and thank-you messages, however I can't see corresponding code to generate them. Either I can change the tests or add code to match.

I think this is what I was working on when I stalled 2 years ago. I suggest opening a new issue, and give some consideration to what the form should be like. Hopefully it's just changing it to match the tests, but might be more complex. Happy for you to also do simple changes here to get the tests passing if you prefer.

I'm not sure that the calculated field Declaration:status is going to work. The value depends on the date, so it would break any caching. Also I'm not sure we need it: the function getStatus() is fine for most purposes.

New issue. We could appropriate cache expiry to the entity. I guess the purpose of it was for display (a computed field being more elegant than hook_entity_extra_field), but I'm open to other approaches to displaying status.

I believe we can rewrite the various functions in DeclarationStorage to work without it - and at the same time be faster, and take an extra parameter of a timestamp. This seems important as we can't always assume the current date is the correct perspective: for example suppose someone adds an order in the past or edits the date or an order. (This one isn't breaking any tests - it's just something I spotted.)

New issue. Sounds like a good idea.

Undefined array key "wrapper_element"

That's an annotation on a commerce checkout pane plugin.

I don't see any sub-module.

It's in the latest version in the gift-aid branch, per my comment on 📌 Migrate basic code Active .

Ah, I caught you out here: the latest version of the code isn't that on the master branch at https://gitlab.com/awakened-heart-sangha/main-site, it's that in the gift-aid branch I gave in the IS: https://gitlab.com/awakened-heart-sangha/main-site/-/tree/gift-aid

Confusing!

#2 done

Sorry, my bad. This actually is fixed in the latest version of the module where there parameters are upcast if not timestamps.

In my use case, I have legacy values in old events where only the start time is known, not the end time. I use the core widget for these events, but the smart date formatter.

jonathanshaw → created an issue.

having a helper method to handle that is very useful, since I need to invoke that logic from both RACH and the RegisterForm. I considered removing it from RegisterForm but that class can be called outside of a context controlled by RACH, for example somebody editing a cart with registrations in it.

Makes sense.

$this->context->getCacheableMetadata()->addCacheableDependency($host_entity);

instead of having to add individual dependencies on the host entity's real entity, and the settings. And extenders of HostEntity can easily add more to it, and all the existing code would pick those additional dependencies up automatically.

Yes, exactly.

I don't think it helps get rid of the constraint dependency mechanism,

I'm not sure why we need the constraint dependency mechanism if we have this. The validator knows what it uses, and adds the appropriate cacheability. Maybe someone has a host entity class that has a custom implementation of hasRoom() that doesn't need the settings entity at all, so hardcoding the dependency into the constraint is actually unhelpful. And having every validator have to add the host entity as a cacheable dependency if the validator calls a host entity method, is repetitive but it's only one line and a nicely explicit. Maybe I'm missing something bigger.

One thing I want to explain here, because it's a bit odd, is how the 'context' for a Gift Aid declaration works.

There's a 'context' dynamic (i.e. can reference any entity type) entity reference (DER) field on the declaration entity type. This is for answering the question "who" is making the declaration.

The reason why this uses DER is that as this is contrib-land we don't know in principle how other sites are choosing to build their database of people. For example, they might be using Drupal's "user" entity type, but maybe they have some (Drupal or external) CRM system and they want to track gift aid declarations against the "contact" entity type from that.

The 'context' reference field allows a declaration to be associated with a particular entity, but allows for further chains of entity reference to be built.

A complex example explains the need more :
- an anonymous customer makes a donation and ticks gift aid at checkout. The context is the commerce order.
- at the end of checkout, they create a user account, which then gets set as the customer on the order.
- the site is using a CRM contact entity type as the actual canonical focus of their Gift Aid system, and the new user is linked up to an existing CRM contact with the same email

The problem is: how do we link the gift aid declaration with the CRM contact? The solution I've created is to use an event as part of the process of querying for declarations. You use web/modules/ahs/gift_aid/src/GiftAidRelatedContextsManager.php to find contexts that relate to other contexts, so that the order declaration can get added in when you are seeking declarations for the CRM contact.

jonathanshaw → created an issue.

jonathanshaw → created an issue.

jonathanshaw → created an issue.

jonathanshaw → created an issue. See original summary → .

jonathanshaw → created an issue.

If the user's previous state was waitist:
- if their new host has no room off waitlist, they stay on waitlist no problem.
- if the new has has room off waitlist, and autofills, they should be auto-filled as normal
- if the new has has room off waitlist, but doesn't autofill, and there are other registrations on waitlist, they stay on waitlist to prevent themselves queue jumping.
- if the new has has room off waitlist, but doesn't autofill, and there are zero other registrations on waitlist, maybe they should be given a space off waitlist like anyone who freshly registered would?

If the user's previous state was complete, but their new host has no room off waitlist, then their new state should be waitlist, and they should be warned of this.
We could/should warn in 2 different places:
(1) Using PossibleHostEntity::setDescription() to append a message in the listing in ChangeHostController
(2) Adding a warning to the change host forms too?

I don't plan to work on this at this time.

Thanks for creating the module component! I'm going to copy your recent validator memory cache here.

Elsewhere you have decided to not block access to register of the host is not available, for performance reasons. So we're currently inconsistent if we block access to edit using the validation mechanism, but don't block access to register.

I think the best solution might be to do what I suggested in comment 5 of 📌 RegistrationAccessControlHandler should rely on RegistratonHostEntityHandler to consider the validation for edit access Active : to use the host access handler but not the validation mechanism. But when the host access handler checks close and status directly, have it check a global setting also about whether or not those should block access.

Sorry, I created an issue fork before seeing it was assigned to you.

The dependency mechanism is there not just for the settings, but to take care of adding the cacheability. Otherwise every validator would need to repeat its own code to add cacheability for the host entity and the settings entity.

A good consideration. But 📌 HostEntityInterface should implement CacheableDependencyInterface Active is something we should probably do anyway because we have the same need in other circumstances, and it would be a more conventional and no more complex DX if we used it in constraints, without the maintenance burden of the validator dependencies mechanism. Not important, just an observation.

The residual issue I'm aware of here is I'm not sure what cacheability I need to add for $host_entity->isConfiguredForRegistration().

This may be a good example of why 📌 HostEntityInterface should implement CacheableDependencyInterface Active is helpful.

I've rebuilt the MR, hopefully clean this time.

To be honest, I don't really see why we have HostEntity::isEditableRegistration() and its RegistrationisEditable constraint. It would see much more Drupal normal to just check close and status here in RegistrationAccessControlHandler and have hook_registration_access as the customisation mechanism.

It's good to allow the host to have an opinion here. But the host's opinion shouldn't consider the specifics of the registration, only the specifics of the host. Considering the specifics of the registration is the job of RegistrationAccessControlHandler with hook_registration_access() as its customisation mechanism.

The best way for the host to have an opinion is to add these close and status checks into the 'edit registrations' host op. This doesn't worrk currently because of the way we call host operations from RACH as currently we do:

    // The "host" permission grants access if the user can edit the host entity.
    if (($result->isNeutral()) && ($host_entity = $entity->getHostEntity())) {
      $result = $host_entity->access($operation . ' registrations', $account, TRUE);
    }

We could treat the 'update' operation different and use andIf() for that case. Or, we could change the way we call all host operations to use andIf, and have the host handler return allowed by default instead of neutral by default. We have noted that host handlers can't currently block access, which feels like something they should be able to do, so something needs to change here anyway for more than the 'update' operation.

Inside the host handler, we could call the validation in the way that HostEntity::isEditableRegistration() currenly does. But actually that feels pretty eccentric to me. I'd just check the status and close directly in the Host access handler and not use validation for this.

Something that did come up in this MR which may be 3.4 relevant is that $host_entity->getSetting('status') or $host_entity->getSetting('capacity')is not the best design pattern. We need to move towards $host_entity->isEnabled() and $host_entity->getCapacity(). In some sense the settings storage is implementation we don't need to expose. This allows the host entity to do more creative things, like consider scopes.

I mention this now, because we use things like $host_entity->getSetting('status') in the new constraint validators, and I'm wondering whether it would ease our future BC if we addressed it before releasing the validators. I'm not sure it does, but I wanted to flag the question as BC can be such a pain.

One thing is definitely true though: if we added these getters to host entity, we wouldn't need the dependency mechanism in the validator and constraints, which would be a simplification win.

This scope work, and any possible multiple field work, should probably be a totally new version - like 4.x - not deprecating, but starting over. Perhaps a migration path - but not like going from 3.3 to 3.4. A complete BC break. Not wedded to this but feels like we need it. ... this feels like more of a rewrite.

My sense at the moment is that it is actually surprisingly backwards compatible. The HostEntity grows in what it considers, if you give it a scope. But by default, it doesn't have any scope so its behavior is unchanged.

The reason why there's quite a lot of deprecation in the MR here is because
(a) all of the existing events expected a 'host_entity' array key to be passed to them and in order change this to 'scope' I deprecated, but I could equally have just passed 'scope' as well as 'host_entity'; and
(b) once I started deprecating events I got greedy and reworked a chunk of the room/space/capacity logic that I found weird but which is hard to change without deprecating events

You could fairly accuse me of gratuitous scope creep.

Where it will get interesting with BC and new versions is in other aspects of ✨ A plan for Drupal CMS support Active like a new variation entity type and changes to IEF support to play nicely with that. But even there I suspect we might mostly need to change the defaults for new installs, not mess with existing installs.

Let's tidy up 3.4 work and get that release out sooner rather than later. I think we are very close already.

Agreed. It's falling into place nicely.

I want to prioritize QR codes and ticketing in 3.x before attacking scopes - of course you are free to continue scope work as you see fit.

Sadly I shouldn't really let myself spend too much time playing with this. I think good Starshot support is valuable for the adoption of this module, and I'm happy to discuss, but I need to rein myself in.

Agreed. The risk/benefit ratio is poor, and its simpler without. Continuing to checking closed before enabled so that message comes first does seem like a good idea though.

Can you recheck that? It's not happening for me in the Gitlab GUI when I look at the latest !103.

It's easy to be unintentionally looking at an older commit in the Gitlab GUI

OK, I've refreshed the branch, and got the code to actually work enough to pass the existing tests.