I'm not really sure we need to set these values at all. If we don't call these methods, Commerce will automatically set these timestamps for us. Here's a patch that just removes this logic.
Here is an updated patch to implement this functionality for 8.x-1.7.
Here is an updated patch that also accounts for the changes to `ContainerAwareEventDispatcher::dispatch()` in Symfony 4.3+.
Patch #5 did not work for me; it evaluates the previous value of the input rather than the current one, so it only works as long as you are not changing the hours.
The root of the problem seems to be in the new formatters introduced in 1.9; here is a patch that targets this specifically. This resolves the validation issue for me, and may resolve other (as yet unidentified) issues as well.