Simple decimals fail to pass validation

Seems points in #291 have been addressed.

+++ b/core/lib/Drupal/Core/Field/Plugin/Field/FieldType/DecimalItem.php
@@ -69,7 +69,7 @@ public function storageSettingsForm(array &$form, FormStateInterface $form_state
-      '#max' => 32,
+      '#max' => ini_get('precision') <= 10 ? 10 : ini_get('precision'),

@@ -79,7 +79,7 @@ public function storageSettingsForm(array &$form, FormStateInterface $form_state
-      '#max' => 10,
+      '#max' => ini_get('precision'),

I think we need a update function here for existing fields as we're changing the maximum so current configuration can be invalid. Not sure what we should be doing with existing decimal fields...

Having come across this issue on 9.5.8, applying the patch from https://www.drupal.org/project/drupal/issues/2230909#comment-14626332 🐛 Simple decimals fail to pass validation Needs work has solved this issue.

Prior to the patch we were unable to set a default value of a trillion (1 with 12 zeroes) on a 15,2 decimal field. After the patch it has saved as expected.

+++ b/core/lib/Drupal/Component/Utility/Number.php
@@ -9,6 +9,71 @@
+      // If the float has less significant decimals than the number we can
+      // guarantee, convert it to a string directly.
+      if (preg_match(sprintf('/^\d+\.\d{1,%d}$/', ini_get('precision')), (string) $number)) {
+        return (string) $number;
+      }
+      // For floats with more significant decimals than the number we can
+      // guarantee, discard those that are not guaranteed.
+      return rtrim(number_format($number, ini_get('precision'), '.', ''), '0');

+++ b/core/lib/Drupal/Core/Field/Plugin/Field/FieldType/DecimalItem.php
@@ -69,7 +69,7 @@ public function storageSettingsForm(array &$form, FormStateInterface $form_state
       '#title' => $this->t('Precision'),
       '#min' => 10,
-      '#max' => 32,
+      '#max' => ini_get('precision') <= 10 ? 10 : ini_get('precision'),

@@ -79,7 +79,7 @@ public function storageSettingsForm(array &$form, FormStateInterface $form_state
       '#title' => $this->t('Scale', [], ['context' => 'decimal places']),
       '#min' => 0,
-      '#max' => 10,
+      '#max' => ini_get('precision'),

+++ b/core/tests/Drupal/Tests/Component/Utility/NumberTest.php
@@ -34,17 +34,107 @@ public function testValidStep($value, $step, $expected) {
+   * @covers \Drupal\Component\Utility\Number::normalize
...
+  public function testNormalize($number, $expected) {
+    $this->assertEquals($expected, Number::normalize($number));
...
+   * Provides data to self::testNormalize().

Checking requirements should go to report/status instead of runtime

If you need properly configured server no reason to scatter this checks all over codebase

Also a test should use permutations for precision configured in field and via ini, I mean missing case when ini_get returns wrong data or it <10

Also this code should be overridable from contrib to allow override math via PHP bc, gmp, decial extensions

@anypost - this important patch has been waiting for 7+ years to get committed. we should not expand the scope. If you want to e.g. be able to use BC math, please open a follow-up issue.

Any more work need to be done here?

#294

and #296 or follow up least opened.

I'm not an expert developer and couldn't read all the comments and patches, but I hope this can help.

I was appalled to see I couldn't save web3 wallet balances to my Drupal site (there a lot of decimals places) even if I could easily define my "balance" field (in my custom entity's BaseFieldDefinition) with precision = 65 and scale = 30 (which correctly creates a column in the database with Type: DECIMAL, Lenght: 65,30)...
These values will come from APIs, but while testing with a manual form input a value of 0.123456789012345678901234567890 would be magically converted to 0.123456789012350000000000000000 in the UI AND database (without validation warnings).
Strangely, the same input given programmatically (I tested with ECA model, ie. plugin, both in plain text and in YAML format) would become 0.123456789012350000000000000000 (last non-zero digit = 6 instead of 5 🧐)

So I checked core/lib/Drupal/Core/Field/Plugin/Field/FieldType/DecimalItem.php myself and I found that I could fix the problem just by commenting lines 122 to 127:

  /**
   * {@inheritdoc}
   */
  public function preSave() {
    $this->value = round($this->value, $this->getSetting('scale'));
  }

Note that scale should be already defined by this (right?):

  public static function baseFieldDefinitions(EntityTypeInterface $entity_type): array {

    $fields = parent::baseFieldDefinitions($entity_type);

    $fields['balance'] = BaseFieldDefinition::create('decimal')
      ->setSettings(array(
        'type' => 'numeric',
        'unsigned' => TRUE,
        'precision' => 65,
        'scale' => 30,
        'not null' => FALSE,
        'description' => t('the decimal integer amount'),
      ))
      ->setLabel(t('Balance'))
      ->setDisplayConfigurable('form', TRUE)
      ->setDisplayConfigurable('view', TRUE);

So why do we want to round at all when we are already forcing the precision and scale in the field (storage) definition?
From https://www.php.net/manual/en/function.round I immediately read

round — Rounds a float

,
but this is a decimal!!

Then shouldn't that be just an optional Manage Display's setting for decimal fields?
Also, why isn't the round function called for float number fields (FloatItems.php file in the same folder) instead?

Also not a dev, but I think I can at least clear up things a little.

First, this is not a decimal (as strange as it may sound). Your number is a float (mathematically) and SQL stores it as an exact data type DECIMAL with a given precision (which in fact is a float). There is an explicit FLOAT data type in SQL, but that is not guaranteed to be exact.

The basic underlying problem is that computers "think" binary and not decimally. Representing an arbitrary number with lots of places after the decimal point will inherently produce errors (depending on which number it is, as some ("machine numbers") map cleanly to binary base and some do not. That's why people here see this only with select numbers). See https://en.wikipedia.org/wiki/Round-off_error for details.

The internal representation of a number 0.123456 might be 0.123456789 because of rounding errors. That's why rounding to the exact given scale does make sense. It's no actual rounding of the given number in the human sense. It just makes sure the number is exactly the one entered and has exactly that many decimal places as configured.

Does the patch in #2230909-292: Simple decimals fail to pass validation → not fix it for you? That would be an important information.

I have made the changes as per comment #294. And write a update function in Field.install file but i am not sure where to place it, currently i have placed Field.install here core/lib/Drupal/Core/Field/Field.install. Please review it and give feedback if any. Thanks

patch #305 applies to D10.2 but it does not work.

I tested a default decimal field (10,2) and a large decimal field (14,2)

Problem 1: Both act like a standard field with 10 digits.
Problem 2: the decimal separator is counted as one digit!
Problem 3: There used to be larger numbers possible than 14.

Please see attached screenshots.

I just realized that #305 is just an addition.

#292 does not apply to 10.2

мой любимый тикет на друпалорге

Rerolling #292 for Drupal 10.2

#309 helps to use large numbers. Thank you very much!!!! We can upgrade to 10.2!

If we get out of bounds there is just a nasty crash error. Not a warning in the gui.

1,123,123,123.00 - just ok.
123,123,123,123 -> crash

The website encountered an unexpected error. Try again later.

Drupal\Core\Entity\EntityStorageException: SQLSTATE[22003]: Numeric value out of range: 1264 Out of range value for column 'field_kostenschaetzung_value' at row 1: INSERT INTO "node__field_kostenschaetzung" ("entity_id", "revision_id", "bundle", "delta", "langcode", "field_kostenschaetzung_value") VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5); Array ( [:db_insert_placeholder_0] => 6479 [:db_insert_placeholder_1] => 19995 [:db_insert_placeholder_2] => fue_projekt_kosten [:db_insert_placeholder_3] => 0 [:db_insert_placeholder_4] => en [:db_insert_placeholder_5] => 123123123123 ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (line 817 of core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

Drupal\Core\Database\StatementWrapperIterator->execute() (Line: 44)
Drupal\mysql\Driver\Database\mysql\Insert->execute() (Line: 1400)
Drupal\Core\Entity\Sql\SqlContentEntityStorage->saveToDedicatedTables() (Line: 971)
Drupal\Core\Entity\Sql\SqlContentEntityStorage->doSaveFieldItems() (Line: 718)
Drupal\Core\Entity\ContentEntityStorageBase->doSave() (Line: 486)
Drupal\Core\Entity\EntityStorageBase->save() (Line: 806)
Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (Line: 352)
Drupal\Core\Entity\EntityBase->save() (Line: 270)
Drupal\node\NodeForm->save()
call_user_func_array() (Line: 129)
Drupal\Core\Form\FormSubmitter->executeSubmitHandlers() (Line: 67)
Drupal\Core\Form\FormSubmitter->doSubmitForm() (Line: 597)
Drupal\Core\Form\FormBuilder->processForm() (Line: 325)
Drupal\Core\Form\FormBuilder->buildForm() (Line: 73)
Drupal\Core\Controller\FormController->getContentResult() (Line: 39)
Drupal\layout_builder\Controller\LayoutBuilderHtmlEntityFormController->getContentResult()
call_user_func_array() (Line: 123)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 627)
Drupal\Core\Render\Renderer->executeInRenderContext() (Line: 124)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext() (Line: 97)
Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 181)
Symfony\Component\HttpKernel\HttpKernel->handleRaw() (Line: 76)
Symfony\Component\HttpKernel\HttpKernel->handle() (Line: 58)
Drupal\Core\StackMiddleware\Session->handle() (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle() (Line: 106)
Drupal\page_cache\StackMiddleware\PageCache->pass() (Line: 85)
Drupal\page_cache\StackMiddleware\PageCache->handle() (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle() (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle() (Line: 36)
Drupal\Core\StackMiddleware\AjaxPageState->handle() (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle() (Line: 704)
Drupal\Core\DrupalKernel->handle() (Line: 19)

Same experience as maxilein.

The patch fixes our problem for storing correct numbers.
But if we try and enter bigger numbers we get a nasty WSOD and some SQL out-of-bound errors. Not a warning in the interface.

@maxilein, @weseze There is another issue and patch for the exception thrown when the entered value is too big: https://www.drupal.org/project/drupal/issues/1003692 🐛 PDOException: SQLSTATE[22003] after entering large value in integer and decimal field Needs work

💥 Friends!
🎉🎉🎉🎉 10 years to this wonderful issue! 🎉🎉🎉🎉
🔥 Created 2 Apr 2014 at 00:19 MSK

Congratulations to everyone on this important event, especially parent #1 @exlin

☀️ Good day, everyone.

I guess that's supposed to be sarcasm. Seems that we need to take 309 and add the test from 305 and review against that. A merge request would probably be helpful. The hook_requirements in 296 hasn't been addressed either I don't think.

i am seeing the same problem in D10.2.5...any fixes yet?

I can sadly also confirm this core bug which basically makes using decimal fields with a default value impossible.

https://git.drupalcode.org/project/drupal/-/merge_requests/579 still has their target on 9.2.x and can be removed safely.

New MR targeting 11.x can be found here: https://git.drupalcode.org/project/drupal/-/merge_requests/8282.

[...] and add the test from 305 and review against that

I think you meant the update hook provided in #305? Unfortunately, the update hook from #305 is pretty much a template implementation and won't work in its current state. I'll give it a shot.

The hook_requirements in 296 hasn't been addressed either I don't think

I'd agree with @pwolanin in #298:

@anypost - this important patch has been waiting for 7+ years to get committed. we should not expand the scope [...]

Any further work on this should be done in a follow-up issue.

Looks like everything works as expected! Let's wait for the tests to succeed.

Here is a static patch of the current MR for the time being (which also applies to 10.2.x, 10.3.x and 10.4.x).

Tests are all green now! Ready to get reviewed. 👍

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Nice work! Just gave it a try and at least I can now save the numeric fields I wasn't able to safe before! :) YAY

Sadly the update hook still needs some work:
The SQL storage cannot change the schema for an existing field (field_example in commerce_product entity) with data.

Alright, the field storage should be updatable now, through the in 10.3.x newly introduced "updateFieldStorageDefinition()" method!( https://www.drupal.org/node/3397892 → )

Please review (no idea why the review bot set this to needs work, it definitely applies on 11.x).

Here is the new static patch (Update hook won't work for Drupal < 10.3.x).

Probably because there is a 9.2 MR and a mix of patches when probably should just be a single MR and no patches

Thanks for all the sudden speed!!

I'd love to help by testing it.

Why are the automated tests only limited choices ( e.g. 10.1.x) ?
Is this already testable?

@maxilein yes, this is testable if you are using 10.3.x! Just apply https://www.drupal.org/files/issues/2024-06-04/2230909-325.patch → through composer! :)

@smustgrave probably... I don't have the permission to close https://git.drupalcode.org/project/drupal/-/merge_requests/579, unfortunately. Anybody with proper permissions, feel free to close it in favor of https://git.drupalcode.org/project/drupal/-/merge_requests/8282.

Sorry. I am still on 10.2
I don't have 10.3 yet

The HOOK_field_widget_form_alter has been deprecated, I think the issue summary needs update.

function hook_field_widget_single_element_form_alter(&$element, \Drupal\Core\Form\FormStateInterface $form_state, $context) {
  $field_definition = $context['items']->getFieldDefinition();
  if ($field_definition->getType() == 'decimal') {
    $element['value']['#step'] = 'any';
  }
}

https://www.drupal.org/node/3180429 →

No need to set this to "Needs work" if the workarounds in the issue summary are starting to get outdated. The workarounds are not really part of the issue, and are only there because of our failure to fix this in the last 10 years.

Cleaning patches for clarity

May need a rebase as the test-only feature is failing to run.

Left some small comments.

Appears to have consistent Unit test failures.

But test-only feature is now running and showing coverage!

Just pitching in with my report: after patching Drupal 10.3 I have no more issues with large numbers with large decimal scales. Thanks everyone!

MR !8282 applies to D10.3.1.
But the new max value seems a bit low.

I have a decimal field with precison 20 and the update_hook fails on it.
Is there a way to force the update?

drush updb
 -------- ----------- --------------- ----------------------------------------
  Module   Update ID   Type            Description
 -------- ----------- --------------- ----------------------------------------
  field    10001       hook_update_n   10001 - Update existing DecimalItem
                                       fields to match the new maximum value.
 -------- ----------- --------------- ----------------------------------------


 Do you wish to run the specified pending updates? (yes/no) [yes]:
 > y

>  [notice] Update started: field_update_10001
>  [error]  The SQL storage cannot change the schema for an existing field (field_with_prec_20 in node entity) with data.
>  [error]  Update failed: field_update_10001
 [error] Update aborted by: field_update_10001
 [error] Finished performing updates.

I also tried to use this module: https://www.drupal.org/project/precision_modifier →
But it is not working (I opened an issue https://www.drupal.org/project/precision_modifier/issues/3460201 🐛 D10.3.1 on decimal field with precision 20 no check appears Active )

Is there a way to force this update?

I am confused about the max size of the decimal numbers.
Why do we limit fields to 14?

Here https://www.drupal.org/docs/7/api/schema-api/data-types/decimal-numeric → it says:

MySQL mappings in the schema.inc file:

'numeric:normal' => 'DECIMAL',

We use DECIMAL data type to store exact numeric values, where we do not want any rounding off, but exact and accurate values. A Decimal type can store a Maximum of 65 Digits, with 30 digits after decimal point.

If Maximum Digits (or Precision) is not defined, It defaults to 10 and if Scale is not defined, it defaults to 0.

All basic calculations (+, -, *, /) with DECIMAL columns are done with a precision of 65 digits.

MySQL also says it uses 65 digits.
https://dev.mysql.com/doc/refman/8.4/en/precision-math-decimal-character...

@maxilein Re: #336: Have you read all the comments above? That would be important.
Especially see #200!

My impression is, that this change is made to align the database schema max with the max php precision value?

Floats are complex, so this should definitely please be checked by someone who digged into this in the recent past. E.g. see https://stackoverflow.com/questions/14587290/can-i-rely-on-php-php-ini-p...

Thank you Anybody. I really missed the setting in the php.ini

My question was: if the database gives us a specified DECIMAL column with a precision of 65 digits - why would we limit ourselves to 14?!
That does not make any sense. Even if it is the default precision limit of php.

Maybe we should think of a informative check on the Status page - just to let anybody more easily understand the situation:

php DECIMAL precision: 14 
This value can be increased in php.ini: precision = 14 ). The max float precision of your system is: 1.7976931348623E+308

echo "The max float precision of you system is: " . PHP_FLOAT_MAX;

Testing the php.ini precision value with larger precision.

I have a decimal field with a precision of 20 scale 2.
(It dates back to when there was no limit by this patch). It has worked ever since.

When I use the field with a default php precision of 14:

I can enter numbers within the limits and I can enter something after the decimal point which other than zero.

When I use the field with a default php precision of 24:

I can add save large numbers according to the 20/2 settings: 123.123.123.123.123.123,00

BUT I can not enter other digits other than zero behind the decimal point:
eg. 123.123.123.123.123.123,01 -> is not a valid number.
Even when the numbers are really small (e.g. 890.34)

Another observation when precision on the site is set to 24 (instead of the default 14). It probably has nothing to do with decimal fields, but maybe someone of you can think of other potentially unsafe situations.

False Status Message: Incompatible with PHP 8.3.6: https://www.drupal.org/project/diff/issues/3462791 🐛 False Status Message: Incompatible with PHP 8.3.6:Diff Active

the comparison from 8.3 in the info.yml file of the module is compared to a float and fails.
I have not yet get a feedback of the module developers. So I can't say how they do that comparison.

Attached is the same patch as in #325 but with unix line endings.
Needed to be able to use the patch with an older version of the unix patch command line tool.