Enhancements to declaration entity

For discussion:

J: AFAICS "advised of liability" is a requirement of any valid declaration at the point of declaration.
A: I agree, but still two reasons.

• The audit process requires a record that this occurred. If we make a box for it, and the staff have to tick it, it makes them much more likely to remember, and should be more convincing to the auditor.
• We could receive an email saying "please add gift aid to my donations". We would naturally create a declaration at this point that is not yet valid because not advised. We could send a written record to remedy that.

J: Oral seems to require a written record unless you have an audio recording of it? That's why I had this on the DeclarationType.
A: True, but this brings a risk that we force the site owner to double the number of declaration types: each one has a variant for oral and another for non-oral. I feel we should leave the site owner free to choose how they use declaration types. E.g. they might instead want declaration types to correspond to donation types: one for commerce, one for direct debit, one for a sponsorship form etc. If we have a field for "method" the software can calculate when a written record is required automatically (it would also be true if we are missing some information).

J: I was thinking to use bundles here [for method and evidence fields], with different configured fields in different bundles. We should provide config for the common circumstances, but allow sites to modify further for their use case. As the fields aren't used programatically, it seemed better to allow configuration to do the work here?

I feel we should leave the site owner free to choose how they use declaration types. E.g. they might instead want declaration types to correspond to donation types: one for commerce, one for direct debit, one for a sponsorship form etc. If we have a field for "method" the software can calculate when a written record is required automatically (it would also be true if we are missing some information).

Evidence is required by HMRC - if we have a base field we can enforce that. We could make evidence a reference to a media type, and add a field to say what it is evidence for. Then we could write some code that checked that the current changes that are being saved have matching evidence.

The debate is good, but I stand by my position.

The key point is that what amounts to evidence, and what is a good UI for entering that evidence, is not something we can know. Only the charity will know its existing systems for capturing and storing evidence & how to integrate with them, and what the predominant use cases are for data entry in its situation and how best to optimise UX for them.

We can't know that a media reference field is the best UX for any or most use cases. For example, when we create an email bundle I will suggest that we put fields "email address" and "email text received" on it as that's the best way to create a gift aid trail for an email.

The bundles should focus on optimal UX for different declaration scenarios.

I'm imagining we should provide configured (so optional and alterable) bundles:
- commerce (i.e. checkout)
- web (i.e. self declare by web UI)
- email (staff declare)
- phone (staff declare)
- face to face (staff declare)
- letter (staff declare)
- paper form (staff declare)

The question of when to send a written confirmation is rather complex. I do agree it needs careful discussion. See 🌱 Product requirements Active 3.2.
It looks like site builders should be able to require a written confirmation on a bundle, disallow a written confirmation on a bundle, or make it optional as staff specify.

'Advised of liability' is an interesting one. HMRC seem quite hot on it as part of an audit trail. I have some concern that the fact that e.g. a checkout is configured to show the liability message could provoke a query - can we prove that on such and such a date the user actually saw a liability message and what exact message did they see? It makes me wonder whether it's good even for commerce, web, paper form contexts to have a field that we can set to indicate that the liability advice was given at time of declaration. I'm wondering if even if we prefill it and don't allow staff to untick it on some bundles, it might still be useful at audit and 'look more convincing'.

The debate is good, but I stand by my position.

You make excellent points, I agree.

It looks like site builders should be able to require a written confirmation on a bundle, disallow a written confirmation on a bundle, or make it optional as staff specify.

Yes, I agree. I would phrase it like this:
- Some definitely need validation as per the regs (oral)
- Some definitely don't because we know that we showed the info (web self-declaration)
- Some need it conditionally, depending on 'Advised of liability' (and we can write clear advice to staff for filing out 'Advised of liability')

can we prove that on such and such a date the user actually saw a liability message and what exact message did they see

This might point back towards storing the message on the charity, and making that entity have revisions?

I'm wondering if even if we prefill it and don't allow staff to untick it on some bundles, it might still be useful at audit and 'look more convincing'.

Actually staff shouldn't be creating declarations for the bundles that don't need validation (such as web self-declaration), and we could enforce that (with admin override).

- Some need it conditionally, depending on 'Advised of liability' (and we can write clear advice to staff for filing out 'Advised of liability')

That's a good idea.

For my legacy paper forms scenario, I can create a bundle specific to that scenario where confirmations are always sent, not optional for staff to decide. It means I need a seperate bundle for new (non-legacy) paper forms, but that's acceptable solution for an edge case like this.

Actually staff shouldn't be creating declarations for the bundles that don't need validation (such as web self-declaration), and we could enforce that (with admin override).

Agreed. We could have permission per bundle, but actually I think it's slightly confusing to use the normal Drupal idiom for something that has compliance implications. Better probably to have a bundle setting "Allow declarations of this type to be recorded by other users." because that way we can add more explanatory notes etc in the UI. The permissions UI is not a good place for stuff that has complicance implications.

Please can you review the IS?

I added some text in italics to highlight areas of flexibility or uncertainty.

Sorry to be slow to reply.

Replace "written_record_required" with "method" field with 3 states: oral; writing; web. The corresponding state of "written_record_required" is yes; prompt (staff to validate the evidence and tick a box); no.

I feel unsure here. My thinking on this hasn't fully adapted from the old plan where there where different bundles and so staff would encounter a page where they first selected the bundle with an explanation of when to use which, to the new plan where there may not even be declaration bundles and it's the evidence field referencing evidence entities of different type that handles the different delaration methods.

A few thoughts I'm having:

(1) The guidelines distinguish between oral and everything else. And we could actually treat web as a species or written. So maybe the method could just be a boolean is_oral.

(2) How do we create a good UX for staff to select which kind of evidence entity to attach? IEF for example tends to offer just a dropdown with the bundle label, which is not a reliable UX for an important compliance-relevant decision like this. Two possibilities:
(A) use a modal for selecting the evidence bundle when adding evidence. Maybe there's an IEF variant for this.
(B) Reverse the entity creation flow. First the staff choose the evidence bundle from a standard bundle-based entity add page, then they go to a declaration form prepopulated with empty evidence of the chosen bundle.

(3) If we use option B, then we could programatically set the is_oral field depending on the evidence type chosen (if we had an is_oral field on the evidence type, specified by the sitebuilder).

I see that the word record is predominant in 3.10.1 so it's inconsistent! Also A few times it uses "written confirmation". The word "record" is least distinctive, used in many other ways a total of 101 times whereas statement is only used in this context.

"written statement" is used in 3.8 to describe a process for retrospectively validating invalid declarations. "written record" is used in 3.10.1 for what needs to be sent to oral declarants. The two are functionally similar and ideally our work on 3.10.1 should made it easy to support 3.8 in the future, but 3.8 is out of scope for now and a rarer use case. So that's an argument for "written record". "Written confirmation" could be argued to be good because it covers both situations without being overidentifed with either.

DONE Change field "invalid" to "valid". It seems more straightforward, and implies an active choice to demonstrate validity. Optional

Validity is an interesting one. There's 2 main flows in the guidance:
1. An oral declaration is not "effective" if there's no recording until a "written record" is sent. And it its "cancelled" within 30 days of the written record being sent then it is treated as if it's "retrospectively" cancelled and treated as if it were never effective.
2. An declaration with evidentiary or data problems can be deemed "invalid" retrospectively. A charity can "validate" it by sending a "written statement". If undelivered, then treat as "cancelled". If not "cancelled" within 30 days, then the declaration can be considered "valid".

Notably, it's not obviously impossible to send multiple written statements if something weent wrong with the first one, or multiple evidentiary problems were found successively.

So there's 3 states:
1. Invalid/ineffective before written statement sent
2. Pending, will be effective in 30 days
3. No problems, all good

It seems like we might need:
1. An effective_date field that indicates the date from which the declaration can be relied on.
2. A mechanism to set the effective_date to 30 days subsequently when a written record/statement is sent.
3. A way to mark a declaration as cancelled from the moment it was created, being never effective.

Not sure how best to handle all this.

My thinking on this hasn't fully adapted from the old plan

Nor has mine😃. First there was just one declaration entity. Then we added #2, the donor entity, for good reason to share storage of the address, but I've not fully processed the consequences. For example it seems to imply that Commerce-based declarations need to put the user as context rather than the order (else they will each have a separate donor referencing their separate order). If there is no user, perhaps we put the profile?? Anyway I like that the donor entity makes things more concrete and clear - the ContextOverviewController now becomes simply the view page for a donor.

Our newest idea is to add #3 the evidence entity. I had been imagining a new specific new entity type rather than a dynamic reference to any entity. We can make bundles for each evidence type. There's a good reason to separate the types of record (date-based declaration, cancellation, change of address etc.) from the types of evidence (phone, email, web, etc.). But it means another entity: more code, and more complex UI. Declarations would likely always need evidence. Do we force a 2-stage process to create one then reference it from the other? Or do we try and create both together e.g. with inline entity form?

Probably next we need entity #4 the record that isn't a declaration - at least we need somewhere to store the key data e.g. for a cancellation: date of cancellation, and what does it cancel - a specific declaration (maybe relevant later for non-date-based declarations?) or all?

Or we could go back to combining the evidence into the declaration. It's simpler, with the downside of some duplication - we need to recreate the evidence types as bundles also for entity #4. However it's a less important case, and perhaps we don't need so many of them?

1)

The guidelines distinguish between oral and everything else.

In many places yes. However web and written declarations are fundamentally different evidence types for our software. The first is entered by the donor; has automatically generated evidence that references the order, and the displayed UI text; has corresponding code so is locked in the UI or at least can't be deleted or created. The latter is entered by staff, needs evidence such as a PDF, photo or email, and we should prompt staff to verify its various parts.

This page specifies 3 types

You must keep records of all Gift Aid declarations, whether they are written, online or verbal.

"Written confirmation" could be argued to be good because it covers both situations without being over-identifed with either.

You make an interesting argument for their being 2 separate cases, but I'm still not convinced.

Firstly I feel that 3 the separate terms aren't as rigorously separated as you suggest, and in fact are potentially used interchangeably. 3.8.2 talks about a written statement for an oral declaration. This page says

You must send or give donors a written confirmation of a verbal declaration.

This implies irrespective of whether you have a recording (which was the assumption I made, but I agree 3.10.1 suggests only if lack of evidence). If oral needs a confirmation only if the evidence is missing, then it seems identical to written??

Secondly, I feel that the behaviour in the 2 cases (3.8.1 or 3.10.1) is effectively identical. The staff members evaluates if the declaration is inherently valid based on the evidence (the criteria might vary slightly for oral, but that's all). If not then it needs needs written confirmation. Any cancellation within 30 days of sending the confirmation is backdated. A valid declaration could later be reclassified e.g. based on an audit.

Validity is an interesting one.

I agree that "valid" and "written statement" could be combined into a single field. Possibly there's an extra state: fatally invalid, and we don't even intend to try and validate it with a written record (this state is currently possible with the 2 separate fields). Possibly we don't need the pending state, if we make the assumption (as per the requirements 3.3) that any declaration less than 30 days ago can be edited to correct a mistake => we wait until May 5th before putting together the annual gift aid claim.

I do wonder if the created date is so important for so many audit etc purposes that it would be useful to keep it as a field (so it s easily available in views, data exports, etc.).

Really? Perhaps you mean the declared date, which I agree is important and we do have a field for. Created date is just the moment that staff found time to process that pile of forms that came in last week. I propose to remove it because people will get it mixed up with declared date😃.

My proposals

1. I will try inline entity form for creating the evidence entity. At first glance, it might seem like more work as we increase the number of entities, however I feel it could actually go the other way: 2 simple entities with a specific purpose could be easier than one huge complex one.
2. Keep 3 states oral, written, web. I like that you try to simplify, but I don't see how to make it work without all 3.
3. Go with your suggestion of "written confirmation". I predict that the same code can be used for both cases, however one of them is out-of-scope anyway so it isn't a problem for now if that isn't 100% true.
4. Add your effective_date field, which I like.
5. Create a multi-state validity, merging the two fields, good idea. This becomes very close to the status field, so I'm attracted to drop status as a separate computed field (solving the caching problem of 🐛 Declaration:status problem Active - the other option of adding custom caching could be expensive and difficult to debug).
6. I feel inclined to avoid making the effective date be part of the validity state as that would require a cron hook, batching, and possible error cases. Instead we can cover this at the report stage (as per requirement 4.1), adding a check on effective date being in the past to the report view.

For example it seems to imply that Commerce-based declarations need to put the user as context rather than the order (else they will each have a separate donor referencing their separate order). If there is no user, perhaps we put the profile??

I think we better open an issue to explore this. It indicates that we have to be careful to make sure that we don't lose somehting of what the context system offered when we introduce donors.

#3 the evidence entity. I had been imagining a new specific new entity type rather than a dynamic reference to any entity. We can make bundles for each evidence type

Yes.

Probably next we need entity #4 the record that isn't a declaration - at least we need somewhere to store the key data e.g. for a cancellation: date of cancellation, and what does it cancel - a specific declaration (maybe relevant later for non-date-based declarations?) or all?

I'm doubtful we need a record entity type in addition to the evidence entity type. I think revision of the donor and declararion entity types, combined with the addition of new referenced evidence entities, is likely sufficient. This will probably mean things like cancellation need custom forms with a tailor made UX that do bespoke entity manipulation, rather than a simple use of an entity form. But that's ok.

However web and written declarations are fundamentally different evidence types for our software. The first is entered by the donor; has automatically generated evidence that references the order, and the displayed UI text; has corresponding code so is locked in the UI or at least can't be deleted or created. The latter is entered by staff, needs evidence such as a PDF, photo or email, and we should prompt staff to verify its various parts.

I'm not particularly fussed about 2 methods vs 3. But FWIW I don't yet fully share your perspective. Your argument there for me mostly points to the need to have 2 seperate forms, a self one and a staff one. I'm less certain whether that will involve an explicit need to flag web vs written or whether it will be obvious from the fact that declarant = creator.

Perhaps you mean the declared date, which I agree is important and we do have a field for.

Thankyou, you're right.

Create a multi-state validity, merging the two fields, good idea. This becomes very close to the status field, so I'm attracted to drop status as a separate computed field

Agreed. I suspect we need to postpone status until later when we have more clarity over when we need it and what for.

I feel inclined to avoid making the effective date be part of the validity state as that would require a cron hook, batching, and possible error cases

Possibly we will have a computed and a stored validiity (obviously not called the same!).

I will try inline entity form for creating the evidence entity.

I have UX concerns there as per #12.2 above. So by all means investigate, but please don't sink too much time into it. I have found IEF to be one of the most fragile, buggiest, complex and poorly maintained parts of the Drupal ecosystem. So I'm nervous on making this module's complicance critially dependent on significant customisations of it.

Thanks, good points.

I have raised 📌 Figure out contexts and donors Active .

I'm doubtful we need a record entity type in addition to the evidence entity type.

Yes, that would work. I put your comment on 🌱 The problem of changes Active .

I'm less certain whether that will involve an explicit need to flag web vs written

I would use the flag so the web types don't appear in the list for staff to create (they only work as evidence for a self-declaration; even if staff create a commerce order on behalf of a donor they would need to use one of the other evidence types). Also the flag could affect access/validity: staff wouldn't need edit-access for 30 days to correct a data-entry error, hence we could allow claims to HMRC without the delay. It won't significantly affect the development time whether 2 states or 3.

Possibly we will have a computed and a stored validity (obviously not called the same!).

Yes that's a good way to describe it. The stored validity is part of this issue. I agree that the computed one can wait until we use it. Potentially it should output translated UI text. The current getStatus() function is a reasonable starting point.

I have found IEF to be one of the most fragile, buggiest, ...

Thanks for the warning!

I would use the flag so the web types don't appear in the list for staff to create (they only work as evidence for a self-declaration; even if staff create a commerce order on behalf of a donor they would need to use one of the other evidence types). Also the flag could affect access/validity: staff wouldn't need edit-access for 30 days to correct a data-entry error, hence we could allow claims to HMRC without the delay.

FWIW neither of those arguments seem right to me.

You don't need the flag *on the declaration entity* to remove web types in the list for staff. You need it on the evidence bundle. The staff form can show all the bundle types of evidence that have the bundle flag. The self-declare form doesn't overtly mention evidence bundles at all.

The validity argument is worth considering, but that seems like what we have the claimable_date field onthe declaration for. The self-declare form can just set the claimable date to today, the staff declaration recording form sets it to 30 days from today.

Yes, yes, yes and yes😃.

Absolutely we are talking about what was originally declaration type, and will become evidence type in ✨ Create evidence entity Active (this issue is complex enough already so I just split that out). Sorry this issue has been a bit slow to adapt to the introduction of the evidence entity. I believe that every reference to declaration types will become evidence types, and probably we will no longer even have declaration types.

You are right, the differentiation mostly works better coming from the different behaviours of the staff vs donor forms. Probably the only time the code will explicitly check for "web" is to hide some bundles from staff. We can also usefully display the web aspect in the UI.

I would prefer to get something a bit more complete then get you to review and test the whole module thoroughly (rather than you reviewing each issue one-by-one in a slightly arbitrary stopping point).

OK for me to commit and continue working please? Probably ✨ Create evidence entity Active next.

Sure

Thanks