Skip to content

round/3 ignores the caller's rounding mode for values wider than the context precision #247

Description

@tomciopp

Summary

Decimal.round/3 rounds twice for any value whose coefficient has more
significant digits than the context precision. The first rounding uses
Context.get().rounding; only the second uses the mode passed to round/3.
When the two disagree the result differs from a single correct rounding,
including a :down round returning a value larger than its input.

Reproduction

Under a reduced-precision context:

Decimal.Context.set(%Decimal.Context{precision: 3})

Decimal.round(Decimal.new("0.4995"), 1, :down)
#=> Decimal.new("0.5")
#   expected Decimal.new("0.4") — :down truncates toward zero

The context's :half_up carries 0.4995 to 0.500, and :down then
truncates that.

The leak runs in both directions, truncating first under :floor discards
the nonzero digits :ceiling needs to see:

Decimal.Context.set(%Decimal.Context{precision: 3, rounding: :floor})

Decimal.round(Decimal.new("0.4001"), 1, :ceiling)
#=> Decimal.new("0.4")
#   expected Decimal.new("0.5")

Reachability

At the default context the parse limit (max_digits: 34) and the precision
(34) coincide, so an over-precision literal never survives Decimal.new/1:

Decimal.new("0.49999999999999999999999999999999995")
** (Decimal.Error) : number parsing syntax: "0.49999999999999999999999999999999995"

That check only guards the parsing path. Decimal.new/3 performs no digit
count, and a driver decoding a numeric column builds the struct from the
binary wire format rather than parsing a string.

coef = String.to_integer("4" <> String.duplicate("9", 33) <> "5")

Decimal.round(Decimal.new(1, coef, -35), 1, :down)
#=> Decimal.new("0.5")
#   expected Decimal.new("0.4")

The practical exposure is code that deliberately selects a non:half_up
mode :down for truncation, :half_even for banker's rounding,
:floor/:ceiling for directional rounding. Those are the modes that disagree
with the :half_up first pass, and the ones financial code chooses on purpose.

Cause

round/3 normalizes its input, and normalize/1 applies the context on both
of its non-zero branches. context/5 rounds the coefficient to
ctx.precision under ctx.rounding — so by the time do_round/5 sees the
value and applies mode, it has already been rounded once under a mode the
caller never asked for.

Not a regression

Long standing, not introduced by #240 or #241.

I have a fix, but would like guidance first

I have a working change locally, and am happy to send it as a PR,
but it runs into two questions that require a decision.

1. Is the current behavior intended? If round/3's contract is "apply the
context, then round to places", then this is documentation rather than a bug
and should be left alone. My reading is that it should be "round to places
under mode", with the context applying to the result like it does for every
other operation.

2. What should round/3 signal? The natural fix stops the input from going
through the precision pass, which also stops it setting :inexact/:rounded
for an over-precision input. Note round/3 doesn't currently signal for the
digits it discards Decimal.round(Decimal.new("1.234"), 1) sets no flags
so the flags fire today only as a side effect of input width. The options:

  • signal for neither (what the minimal fix does)
  • signal whenever do_round/5 discards nonzero digits, which means threading
    that back out of do_round/5, a wider change, but arguably the correct
    one given round/3 is the operation closest to the spec's quantize
  • leave the flags as they are, if the current behavior is depended on

Side Note:

The exponent limits have to stay applied to the input. do_round/5 scales the coefficient
by pow10(exp - target_exp) when exp > target_exp, which is unbounded for an
exponent built through new/3. Simply replacing normalize/1 with a
context free zero strip would let Decimal.round(Decimal.new(1, 1, 10_000_000), 0)
build a ten-million-digit coefficient before the context discarded it.

Let me know what you think.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions