Images getting deleted on import

I've been puzzling over an issue that I've had with my Migrate script that takes an XML file and imports them as Drupal Commerce Products. (Yes, I'm using commerce_migrate for this, feel free to move this if you feel it's appropriate.) What's happening is that I have two image fields attached that aren't handled by the migration, and on import both of these fields are being emptied.

This happens even though we have preserve_files set:

$this->addFieldMapping('field_image:preserve_files')->defaultValue(TRUE);
$this->addFieldMapping('field_product_image_listing:preserve_files')->defaultValue(TRUE);

The only way I've found around this is use prepareRow() to load the old entity, pull the File ID out, and to use MigrateFileFid to re-add the File ID to the row. However, this is detrimental to performance.

$this->addFieldMapping('field_image', 'field_image_fid');
$this->addFieldMapping('field_image:file_replace')->defaultValue(MigrateFile::FILE_EXISTS_REUSE);
$this->addFieldMapping('field_image:preserve_files')->defaultValue(TRUE);
$this->addFieldMapping('field_image:file_class')->defaultValue('MigrateFileFid');    

$this->addFieldMapping('field_product_image_listing', 'field_product_image_listing_fid');
$this->addFieldMapping('field_product_image_listing:file_replace')->defaultValue(MigrateFile::FILE_EXISTS_REUSE);
$this->addFieldMapping('field_product_image_listing:preserve_files')->defaultValue(TRUE);
$this->addFieldMapping('field_product_image_listing:file_class')->defaultValue('MigrateFileFid');    

prepareRow():

$original_product = commerce_product_load_by_sku($row->xml->ProductCode);

$row->field_image_fid = NULL;
$row->field_product_image_listing_fid = NULL;

if ($original_product) {
  if (isset($original_product->field_image[LANGUAGE_NONE])) {
    $row->field_image_fid = $original_product->field_image[LANGUAGE_NONE][0]['fid'];
  }
  if (isset($original_product->field_product_image_listing[LANGUAGE_NONE])) {
    $row->field_product_image_listing_fid = $original_product->field_product_image_listing[LANGUAGE_NONE][0]['fid'];
  }
}

There must be something I'm missing, right? I've tried this with Migrate 2.4 and 2.x-dev from today, and haven't had any luck either way.

(Of course there are still the performance issues, and all the segfaulting when run from Drush, but that's another story...)