Support other evidence-based practices for learning

This is actually specific to an activity-answer plugin, some of those (basically all automatically scored), can provide feedback to all possible answers or answer ranges (for example if there was a math question what's 2+2 and a number input field, feedback may depend on ranges of the given answer and if it's a multiple choice, feedback should be bound to all possible answers and displayed depending on user choice with different possible scenarios, free text plugins could recognize some patterns and provide feedback basing on that).

In the case of radios, we don't even need an additional button as feedback can be displayed with AJAX when a choice is made, in case of a multiple choice, there should possibly be a "check" button that is replaced by next/corrctt answer.

Due to the above, I don't see a possibility for a general API or even generic feedback storage, all depends on the actual plugin behavior and requirements.

So: try to start creating immediate feedback activity-answer plugins (a few basic ones could even go to another sub-module in the LMS core similar to lms_answer_plugins and more complex ones to ecosystem modules, we can decide case-by-case), if the LMS core API needs to be extended further for this purpose, please add feedback here.

Thanks @graber -- I'll start playing around with some plugins and see what I can come up with.

Do you see a separate set of plugins that incorporate immediate answer checking and feedback, which would eventually be a duplicate set of all the non-feedback plugins? Or should each existing and future plugin include both options of giving feedback or not? It seems like the second option would be more efficient overall.

A more architectural question is whether each question in a course should offer the individual option of giving immediate feedback, of if an entire course needs to be one way or the other.

Do you see a separate set of plugins that incorporate immediate answer checking and feedback, which would eventually be a duplicate set of all the non-feedback plugins? Or should each existing and future plugin include both options of giving feedback or not? It seems like the second option would be more efficient overall.

I don't think the second option would be more efficient because it requires additional storage that would hold empty columns (or even empty data tables) in case when feedback will be off. Also, some plugins (the teacher - evaluated ones like fulltext answer) just don't allow any immediate feedback so will not have their feedback counterparts.

A more architectural question is whether each question in a course should offer the individual option of giving immediate feedback, of if an entire course needs to be one way or the other.

- by question type - if it has a plugin that gives immediate feedback - feedback will be provided.

We should probably start with a single plugin that gives feedback (True/False as it's the most simple one?) and see how it works.

Of course, on a specific platform you can have only questions with feedback and automatically evaluated configured if you want no teacher interaction, no grading etc. Course creators will then not be able to create any learn-grade-pass course workflows but only non-graded where scores will serve some other purpose.

Ok I will dig into creating a True/False plugin with feedback and see where it takes us. 😃

As far as course creators being able to have graded or nongraded courses on a given platform, for my application at least, it would be more ideal if that could be a per-course setting that would be chosen by the course creator. But maybe that can just be dictated by the choice of question plugins that are used in the course.

@arvana I had to adapt the API a bit so we have form state available in activity-answer plugin answeringForm method for this to work fine.
Now you have an example True/False with feedback plugin: https://git.drupalcode.org/project/lms/-/blob/1.0.x/modules/lms_answer_p...

@graber this is great, thank you. The code looks really clean and should be a good base to work from. A few thoughts so far:

1. We need separate feedback fields for each available answer, and some logic in the plugin for which feedback to display based on the answer entered. So if a student gets an answer right, they get positive feedback that reinforces their answer, if they get it wrong they get feedback that gets them to think more about the question. For true/false questions that would just be two feedback fields, if it's a multi-select question it would have two per answer.
    
2. The plugins should also add a style class to the right and wrong answers when the answer checking is done. On the front end those can then get coloured and right/wrong symbols added etc.
    
3. I can see that the number of plugins is going to get unwieldy pretty quickly. On a small website the developer can easily just include the plugins they want on a platform level, but for an enterprise-level site where different courses may have their own individual needs, I think we should have a way to enable/disable different plugins on a course level.

I'll be playing around with those ideas and see what I can come up with.

Ok please ignore my first 2 points above, I didn't check it out thoroughly enough before I ran my mouth! 😂

Yes, for your #3 - we’ll need to support course bundles so they can have different settings. The API is flexible enough but it means other modules like lms_membership_request (actually can’t think of any other that would be affected at the moment) will have to adapt. I’ll leave that for sometime later.

My first MR for this project. 🙂

For true/false questions, it works to display the answer feedback upon answer selection, but for other feedback plugins there will need to be a 'Check Answer' button. For consistency it seems to me that all of the feedback plugins should use the same button, so I'm proposing that we add it to true/false as well.

Also for the front end we'll want to style right and wrong answers, so I've added style classes for each.

I'm wondering about the data format for answer feedback in other plugins. Having a single feedback field with cardinality of 2 is ok for true/false, but it's a little clunky for course creator UX -- the fields don't have labels and can be reordered. It will get pretty unmanageable in plugins with multiple answers, multiselect for example. Any ideas for how to structure those?

It will get pretty unmanageable in plugins with multiple answers, multiselect for example. Any ideas for how to structure those?

Something like this would probably require an additional field property rather than a separate field?

Agreed, the LmsAnswer field type and widget will need to be extended for that one.

As for the T/F MR by @arvana: https://git.drupalcode.org/issue/lms-3502281/-/jobs/4339487

Let's create child issues for next plugins instead of handling here.

Also thinking how to separate feedback plugins from non-feedback ones but no idea at the moment, because both may use the same base classes so separate modules are not an option.. but maybe if we moved base classes' contents to traits and placed them in the LMS module? Thoughts anyone?

 
I fixed the formatting issues and committed into the MR.

Also thinking how to separate feedback plugins from non-feedback ones but no idea at the moment, because both may use the same base classes so separate modules are not an option.. but maybe if we moved base classes' contents to traits and placed them in the LMS module?

In my opinion, the feedback and non-feedback plugins are mutually exclusive -- any course will want to use 100% one or the other. A course that only gives a grade at the end will not want feedback along the way, and vice versa. So we could have a radio-select course setting with "Course Type" options: Standard (grade at end) or Interactive (feedback throughout).

Rather than having separate plugins for feedback and non-feedback versions of each question type, could we use a strategy class that's set by the course's feedback setting? Then the feedback data storage, editing, and display would only be injected when called for by the strategy class, all in the same plugin as the standard function.

If we extend the LmsAnswer field type and widget, won't that keep feedback data in the schema for non-feedback questions? Instead, could we have a dedicated feedback field that's only added when called for by the strategy class? That would also give us a lot more flexibility for improved display and course-creator UX.

Logistically, I'm still thinking of building all of the feedback plugins separately first. Then they can be repackaged into a better-structured architecture once their functionality is all figured out. I'm hoping I'll be getting much faster with the rest of the plugins!

I'm still very much learning the module architecture, so feel free to let me know if I'm being a fool, I won't be offended (much). 😁

In my opinion, the feedback and non-feedback plugins are mutually exclusive -- any course will want to use 100% one or the other. A course that only gives a grade at the end will not want feedback along the way, and vice versa. So we could have a radio-select course setting with "Course Type" options: Standard (grade at end) or Interactive (feedback throughout).

We can't be 100% sure of that, those are plugins and developers can make any types of them plus site builders can create unlimited combinations of activity field setups and plugins. You can go for a custom solution here and limit available activity types based on some course checkbox.

Rather than having separate plugins for feedback and non-feedback versions of each question type, could we use a strategy class that's set by the course's feedback setting? Then the feedback data storage, editing, and display would only be injected when called for by the strategy class, all in the same plugin as the standard function.

No, because feedback storage depends on actual plugin and can differ a lot, there's no single, ultimate storage that will fit all.

If we extend the LmsAnswer field type and widget, won't that keep feedback data in the schema for non-feedback questions? Instead, could we have a dedicated feedback field that's only added when called for by the strategy class? That would also give us a lot more flexibility for improved display and course-creator UX.

I thought about extending in a separate field plugin, specific to the new multiple select with feedback activity-answer plugin only.

Logistically, I'm still thinking of building all of the feedback plugins separately first.

Plugin-by-plugin development approach is the best way here indeed as those plugins may not have much in common in various cases really and data storage is handled by each plugin individually.

Regarding your MR, All standards corrected but I think that if there's the "check feedback" button, when going back to an answered activity the feedback shouldn't show.
Somehow I still like the no-button version with immediate feedback more so I guess we need a third opinion here ;)

I decided to merge your PR as I know from my past experience it’s the best way to get some feedback from others and we should follow that kind of policy in the future.
Thank you for your input!

Let’s handle other plugins in child issues, leaving this one as “needs work” so we have a place for general considerations.

Thanks very much, @graber! This is a super interesting discussion -- there are a couple of topics to consider here:

Evidence-based approaches to learning

I definitely agree that giving immediate feedback on answer selection is much cleaner in code. But according to the reading I've done in modern pedagogy, there are learning benefits to making answer selection more interactive. I've tried to base the feedback approach on the merits of research:

• Giving students time to think about their choice, and optionally changing it before checking their answer, has been shown to increase metacognitive processes ("I think this is right because...") -- the theory is that it matches real-world learning where we need to commit to decisions.
   
• The "testing effect" of learning and memory being increased by practice tests gives better retention the more interaction there is with a question.
   
• Allowing multiple attempts and getting feedback for each of them promotes learning through mistakes.
   
• Having time to consider an answer before checking it encourages thoughtful first attempts, rather than guess-and-check behaviour.
   
• It may reduce testing anxiety if we avoid every click feeling like a judgement.
   
• Duolingo's approach makes sense for their context because language learning has clear right/wrong answers and questions are atomic/independent. But for a general-purpose LMS that handles diverse subjects, answers may require deeper and more nuanced thinking.
   
• Some plugins, like multi-select with feedback, will definitely need a 'Check Answer' button, and in my opinion it's better UI/UX to have consistency across a plugin type, i.e. a course with all feedback answers will all behave the same way.
   

Data structure

I had originally started out with separate feedback_if_correct and feedback_if_wrong fields, which gives more flexibility for the course creator UI than a single field with cardinality 2. Plugins like multi-select and free-text with feedback will likely need more complex feedback structure. It seems fine with me to keep that purely as a pure per-plugin data structure.

I think that if there's the "check feedback" button, when going back to an answered activity the feedback shouldn't show.

Thinking more about this, I agree. In more general terms, revisiting a question could theoretically display:

• the previously chosen answer (selected/not),
• styling of that answer (right/wrong),
• and the answer feedback (shown/not shown).

Learning science research says that spaced practice with active recall is most effective for learning. Based on that, it seems like the most effective revisit display should:

1. show the previously selected answer (maintains context);
2. not show right/wrong styling initially (promotes active recall);
3. not show feedback initially (increases mental engagement);
4. require student to actively re-check their answer (reinforces learning).

Cool, let's make that small change here, write a plan which feedback plugins will be nice to have in core and proceed in child issues.

Ahh, sorry, just noticed you created 📌 Develop true/false plugin with feedback Active .. So all changes to T/F and new plugins will be handled in child issues from now on :)