Form API AJAX - Dynamic dropdown fields options dependent on another field's values

Quick intro:

I was recently trying to make multiple dropdown fields programmatically using the Form API. The visibility of the dropdowns were variable, depending on the field preceding it. Specifically in my case, I had an entity form which is used to submit research papers. I had a field which contained Programme Areas which were simply taxonomy terms displayed as checkboxes (for multiple selections), each Programme Area had directors assigned to each one. Some areas had only one director, some had multiple. So when a user selected which Programme Areas their paper covered, they would click a button to retrieve the associated Programme Directors and a new field for each Programme Area they had clicked would appear, all containing options for which Programme Director they would like to submit their paper to for that specific programme area. Here is an example of what I needed to achieve:

The issue:

My issue was that I could not work out how (using AJAX) I could get the selected Programme Areas in the AJAX callback, to determine which Programme Area fields to then display.

My first attempt was trying to calculate all of this inside the callback function. So I'd call upon the $form_state to check which Programme Areas had been selected, which I can do fine. Then I'd generate a new field per Programme Area selected, with options of all the area's associated Programme Directors. However, this created an issue in that it was not correct practice of the Form API and upon submitting and using a custom submit handler, the values of those fields were not in the final submitted $form_state.

This bugged me for a long time until I found a workaround, which works fine, but I believe to be unorthodox and possibly does not follow the Form API's logic (but I'd like to hear other developer's opinions).

The solution:

What I basically did was instead of using the AJAX callback to generate these fields, I simply just returned the fieldset which contained all the elements for the Programme Directors fields and generated the fields in the form hook instead.

<?php
function mymodule_form_programme_directors_ajax_callback($form, &$form_state) {
  return $form['programme_directors_elements'];
}
?>

Then, in the hook_form function, I used a foreach method to declare the Programme Directors fields. To determine which Programme Director fields to display, I tried looking in the form state but I couldn't find out which ones had been ticked in the $form_state['values'] array! After a lot of tinkering and searching, I managed to find out that if you look in the $form_state['complete form']['field_programme_area (or whatever field yours is)'][LANGUAGE_NONE] array, you can see what the values of the field are!

This is basically my final code:

<?php


// The fieldset holding all programme director components
$form['programme_directors_elements'] = array(
    '#type' => 'fieldset',
    '#prefix' => '<div id="fieldset_programme_directors_elements">',
    '#suffix' => '</div>',
  );

//Adding a button to run an ajax process to get Programme Directors of the selected Programme areas.
  $form['programme_directors_elements']['programme_directors_button'] = array(
    '#type' => 'submit',
    '#value' => t('Choose Programme Directors'),
    '#limit_validation_errors' => array(),
    '#submit' => array('mymodule_ajax_submit'),
    '#ajax' => array(
      'wrapper' => 'fieldset_programme_directors_elements',
      'callback' => 'mymodule_form_programme_directors_ajax_callback',
    ),
  );

//This if statement is here so the fields will only appear once the button has been clicked
if($form_state['programme_directors_elements']['programme_directors'] == TRUE){
// Get all programme area terms from the vocabulary
    $areas = mymodule_get_all_programme_areas();
    foreach($areas as $area_tid => $area_name){
// Display the field for this area's programme directors only if the area was selected in the field_programme_area field above it.
      if($form_state['complete form']['field_programme_area'][LANGUAGE_NONE][$area_tid]['#default_value'] != NULL){
        $form['programme_directors_elements']['programme_directors'][$area_tid] = array(
          '#title' => $area_name,
          '#type' => 'select',
          // The function here just runs a query to get the programme directors associated with this area's taxonomy ID. It would simply return an associative array with the users ID as the key and their name as the label
          '#options' => mymodule_ajax_callback_query_builder($area_tid),
        );
      }
    }
  }
?>

The submit function:

<?php
/*
 * Submit for the Programme Directors callback
 */
function mymodule_ajax_submit($form, &$form_state) {
  $form_state['programme_directors_elements']['programme_directors'] = TRUE;
  $form_state['rebuild'] = TRUE;
}
?>

Conclusion

In order to get a dynamic field like this, you cannot generate your options/values in the AJAX callback function, as it doesn't seem possible to pass values in the $form_state array from the callback to the form function, if you need to be able to use these values in your submit handler. The workaround is to check these values in the hook_form function and see which values have been set in the $form_state['complete form'] array.

I'm not sure if this is a bug, or just that I had a very specific requirement that needed this sort of workflow.