How can I set exposed filter options programmatically?

I want a views filter with a select widget and I need to set the options programmatically. It is possible to do this in hook_form_FORM_ID_alter(), however, it doesn't really work as desired. This is because the widget does not respond as expected to the option values I set. Instead, Drupal expects the option values to be group numbers. (See attached screenshot.)

If the actual value I am targeting is 123, there needs to be a group with that value. Thus, it's not possible to set the values in $form['my_field']['#options'] in hook_form_FORM_ID_alter().