Migrate Number from float64 to big.Float

[gavv]

Currently, Number always converts its value to float64. In many cases it works well, however float64 can't represent big integer values (larger than 2^62) without precision loss.

Actually, there is no native type that can represent ALL values of int64, uint64, and float64 without loss. However, there is a library type that can do it: big.Float (see https://pkg.go.dev/math/big).

Here are the steps:

• replace underlying value of Number from float64 to big.Float

  • by default, use precision that is sufficient to represent all values of int64, uint64, and float64 without loss

  • allow user to set higher precision in Config (this may be useful if user wants to parse Number from String that contains bigger integer or float)

• rework canonNumber() to return big.Float instead of float64; it should automatically detect precision and create big.Float from any of these types:

  • all native float and integer types
  • type definitions for those native types (e.g. type MyInt int)
  • big.Float and big.Int
  • json.Number

• rework NewNumber constructor: it should accept interface{} instead of float64 and use canonNumber() to convert it to big.Float

• rework all Number operations, so that they will operate on big.Float values (one stored in Number, another returned from canonNumber for operation argument)

• rework Number.EqualDelta and Number.NotEqualDelta: they should accept interface{} instead of float64; this will allow user to pass big.Float instead of float64, when necessary

• rework JSON decoding:

  • find places where we decode JSON: canonValue, Response, WebsocketMessage; configure JSON decoder to use json.Number instead of float64 (see https://pkg.go.dev/encoding/json#Decoder.UseNumber)

  • ensure that canonValue() and other canonization functions does not cause precision loss

• rework Number creation

  • Expect.Number() should accept interface{} instead of float64

  • rework Value.Number() - it should not force float64 type

  • rework String.AsNumber() - it should correctly parse big floats and integers

  • find all invocations to newNumber() that have unnecessary casts to float64 and pass integers instead; e.g. newNumber(a.chain, float64(len(a.value))) can be now replaced with newNumber(a.chain, len(a.value)) because Number now supports integers

• adjust equality checks:

  • find all places that do deep equality checks (e.g. in Object, Array) using reflect.DeepEqual

  • ensure that they will work when original values have different representations for same numbers (e.g. json.Number vs float64 vs big.Float)

  • cover these cases with unit tests

• add new getters for Number:

  • add 4 new methods: AsInt (returns int64), AsUint (return uint64), AsFloat (return float64), AsBig (return big.Float)

  • each method should report failure if underlying value can't be converted to requested types without loss

  • add IsBig (similar to other Is methods, see Implement Number IsInt, IsUint, IsFloat, IsNan #155)

  • mark Number.Raw as deprecated and ask user to use AsFloat instead; in httpexpect v3, when we will be able to break compatibility, we'll undeprecate Raw, but change its return type to big.Float

• adjust DefaultFormatter to handle big.Float values:

  • if value is integer, print it as integer

  • otherwise print it in non-scientific form (see also Add DefaultFormatter.EnableScientific #190)

• adjust tests:

  • add unit tests for Number to check how it works with big integers and floats

  • add unit tests for Value.Number() to check how it works with json.Number and integers that can't be represented without loss as float64

  • rework tests for String.AsNumber() - rework float_precision test, add tests for big floats and integers that can't be parsed with default precision, but can be parsed when setting higher precision in Config

  • cover big integers and floats, and different precisions, in e2e tests (e.g. in e2e_basic_test.go)

[angshumanHalder]

Hey @gavv can assign this to me? I am little busy with switching jobs but this problem seems interesting. Would pickup as soon as I get some time.

[gavv]

Sure, thanks!

[angshumanHalder]

Hey @gavv converting

type Number struct {
	noCopy noCopy
	chain  *chain
	value  float64
}

to

type Number struct {
	noCopy noCopy
	chain  *chain
	value  big.Float
}

will cause Raw() to fail. As it returns float64. The code won't compile unless we comment out the code instead of deprecating it or we can modify it to return float64 by converting it from big.Float. But if we decide to make changes on the function then we don't need to deprecate it. And AsFloat and Raw would be the same exact function. How do you suggest we proceed?

[gavv]

or we can modify it to return float64 by converting it from big.Float

Yes, that's what I was suggesting.

And AsFloat and Raw would be the same exact function

Yes.

---

Yeah, sorry, I was not clear on this. The idea is the following:

• we add AsFloat, it returns float64; we also add other AsXxx methods
• we keep Raw signature untouched, it still returns float64
• code that uses Raw keeps compiling and working, we don't break compatibility
• however we mark Raw as deprecated and users will get a warning that they should migrate from Raw to AsFloat (if they want float64), or to other AsXxx
• we wait for some time, e.g. a few months; during this period, users will hopefully update lib, see deprecation warning, and migrate to AsXxx methods; since it's a warning, not a error, they can do it gradually in comfortable pace
• after a few months, or maybe a year, I plan to release httpexpect v3, which will remove all currently deprecated functions
• in v3, we will change Raw to return big.Float instead of float64; this will break compatibility, but it's allowed in a major version upgrade; furthermore, since Raw was deprecated way before, many users wont be using Raw at this point, and their migration to v3 will be more smooth

This is actually a common approach for breaking APIs: deprecate, wait, then remove or change. Wait period gives users time to reacts to the change without hurry.

[angshumanHalder]

Hey @gavv I have few questions.

1. There are certain deprecated functions that uses newNumber() should those be replaced too?
2. rework JSON decoding:

find places where we decode JSON: canonValue, Response, WebsocketMessage; configure JSON decoder to use json.Number instead of float64 (see https://pkg.go.dev/encoding/json#Decoder.UseNumber)

ensure that canonValue() and other canonization functions does not cause precision loss

For ☝️ do you suggest to have a type assertion wherever the functions are called? Because as far as I can see. All of the canon functions are using interfaces to decode json rather than any specific type.

[gavv]

1. Yep

2. Sorry, I don't understand the question/suggestion, could you elaborate?

[angshumanHalder]

So let's suppose we have this function -

func canonValue(opChain *chain, in interface{}) (interface{}, bool) {
	b, err := json.Marshal(in)
	if err != nil {
		opChain.fail(AssertionFailure{
			Type:   AssertValid,
			Actual: &AssertionValue{in},
			Errors: []error{
				errors.New("expected: marshalable value"),
				err,
			},
		})
		return nil, false
	}

	var out interface{}
	if err := json.Unmarshal(b, &out); err != nil {
		opChain.fail(AssertionFailure{
			Type:   AssertValid,
			Actual: &AssertionValue{in},
			Errors: []error{
				errors.New("expected: unmarshalable value"),
				err,
			},
		})
		return nil, false
	}

	return out, true
}

How do you want it to be modified to use json.Number? Because from what I can see we are using interface{} as input.

Or Am I understanding this wrong? If so can you please explain this?

[gavv]

Ah, I see.

canonValue and other methods use json.Unmarshal to unmarshal object, array, etc. When json.Unmarshal encounters a number somewhere in json, it unmarshals it into float64. This may cause precision loss if the number can't be represented as float without loss (if it's a large 64-bit integer for example).

encoding/json provides an option to unmarshal numbers into json.Number instead of float64. (This need using json.Decoder instead of json.Unmarshal).

We need to use this option everywhere when we unmarshal json, to avoid precision loss. Our maps, arrays, etc. should have json.Number instead of float64.

When the user will create Number for such element, it will construct big.Float from json.Number.