Skip to content

Add randomness and probability distribution providers - #25

Merged
matt-edmondson merged 4 commits into
mainfrom
claude/sharp-cannon-flnpg5
Sep 13, 2026
Merged

matt-edmondson merged 4 commits into
mainfrom
claude/sharp-cannon-flnpg5

Conversation

@matt-edmondson

Copy link
Copy Markdown
Contributor

Adds two provider categories to ktsu.Essentials: IRandomProvider for uniform randomness, and IDistribution<T> — split into IContinuousDistribution and IDiscreteDistribution — for probability distributions with CDF, quantile, survival function, density or mass, moments and sampling.

Randomness

IRandomProvider declares one primitive, NextBytes(Span<byte>), and layers everything else on it as default interface implementations: range-limited integer draws, floating point on the unit interval both half-open and open, Bernoulli trials, Fisher–Yates shuffling, uniform and weighted choice, and sampling without replacement. Range-limited draws carry no modulo bias — Lemire's multiply-and-shift for 32 bits, rejection for 64, since Lemire needs a 128-bit product that isn't available on every target here.

Four implementations:

Package Seedable Thread-safe For
RandomProviders.Native within a framework version no the platform default
RandomProviders.Crypto no, by design yes (stateless) tokens, salts, nonces, anything unpredictable
RandomProviders.Xoshiro yes, portably no fast reproducible simulation
RandomProviders.Pcg yes, portably, with streams no as above, plus independent streams from one seed

Xoshiro (xoshiro256**) and Pcg (PCG-XSH-RR) write the algorithm out, so a seed replays identically on any machine and any framework version — System.Random explicitly does not promise that, and changed its sequence in .NET 6. Their sequences are pinned by reference vectors from an independent transcription of each published algorithm, which makes a change to one a breaking change that the suite catches.

The three stateful providers register as transients rather than singletons, because they are not thread-safe and a shared singleton would be a data race. The stateless cryptographic provider stays a singleton.

Distributions

An implementation supplies the CDF, quantile, support bounds and first two moments. Sampling defaults to inverse transform, the median to the half quantile, and a discrete quantile to bisection over the CDF — at most 31 evaluations even over an unbounded support — so a new distribution is a small class.

Ten ship: uniform, normal, exponential, log-normal and triangular (continuous); Bernoulli, binomial, Poisson, geometric and categorical (discrete). Triangular, binomial and categorical need parameters with no standard default, so they're excluded from AddEssentials() and registered from their own packages, exactly as the composite obfuscator already is.

Two design points worth review:

  • Where a CDF has no closed form it goes through a shared SpecialFunctions file, linked into the four packages that need it rather than placed in the interfaces-only package (following HmacKeyedHashCore and NonCryptoIncrementalHash). Incomplete gamma by series and continued fraction, incomplete beta by continued fraction, and the error function derived from the incomplete gamma rather than approximated separately — erf(x) is exactly P(½, x²), so there is one implementation to keep honest instead of two. The normal quantile is Acklam's rational approximation refined with one Halley step. Binomial and Poisson CDFs are therefore a single evaluation, not a sum whose length grows with the count.
  • SurvivalFunction is declared, not inherited, wherever the tail matters. The default subtracts the CDF from one, which discards the tail to cancellation: the normal survival function at 8σ is 6.2e-16, and 1 - 0.9999999999999994 keeps about one digit of it. There's a test that asserts the difference.

Neither interface has an async tier, deliberately. The others offer one because they front I/O; a draw or a CDF evaluation is a handful of arithmetic instructions, so a Task.Run wrapper would cost orders of magnitude more than the work. Noted in CLAUDE.md so it doesn't get "fixed" later.

Testing

888 tests pass, and the solution builds clean under the ktsu analyzers (Roslyn 5.9 override, per CLAUDE.md). Three kinds of check:

  • Contract, over every implementation: bounds and validation for randomness; for distributions, that the CDF is bounded and non-decreasing, the quantile inverts it, the density integrates to one, the masses sum to one, and the survival function complements the CDF.
  • Reference, against values computed outside this codebase: PRNG sequences from independent transcriptions; normal, binomial and Poisson values from exact or independent computation, including Binomial(2000, 0.5) as an exact rational over 2^2000 where the coefficient itself is ~1e600.
  • Empirical, from a seeded generator: sample moments and deciles against the analytic ones, and a uniformity check on a range that does not divide 2^32, which is what a biased bounded draw would fail.

Two things the tests found and the diff fixes: the geometric closed-form quantile could be one out where the ratio of two logarithms landed a rounding step past a whole number (now corrected against the CDF), and Binomial(2000, 0.5) loses a few digits to cancellation in the log coefficient — documented rather than papered over, with the tolerance set to match.

Accuracy

Stated honestly in the docs rather than rounded up: the incomplete-gamma route carries the normal CDF and quantile to a few parts in 10^13, and the log-space binomial mass to about a part in 10^11 at n = 2000. Both are far better than the alternatives they replace, and neither is full double precision.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NeGiPWLApq21sJZfMyrmdH


Generated by Claude Code

Adds two provider categories: IRandomProvider for uniform randomness, and
IDistribution<T> with IContinuousDistribution and IDiscreteDistribution for
probability distributions with CDF, quantile, density/mass and sampling.

IRandomProvider declares one primitive, NextBytes(Span<byte>), and layers
everything else on it as default interface implementations: integer draws over
a range (unbiased, by Lemire's method for 32 bits and rejection for 64),
floating point on the unit interval both half-open and open, Bernoulli trials,
Fisher-Yates shuffling, uniform and weighted choice, and sampling without
replacement. Nothing here is asynchronous — a draw is a few arithmetic
instructions, so a Task.Run wrapper would cost far more than the work.

Four implementations: Native over System.Random, Crypto over the OS CSPRNG,
and Xoshiro (xoshiro256**) and Pcg (PCG-XSH-RR), whose seeded sequences are
fixed by this code rather than by the framework and are pinned by reference
vectors from an independent transcription of each published algorithm. The
three stateful providers register as transients because they are not
thread-safe; the stateless cryptographic one is a singleton.

IDistribution<T> requires the CDF, the quantile, the support bounds and the
first two moments; sampling defaults to inverse transform, the median to the
half quantile, and the discrete quantile to bisection over the CDF. Ten
distributions ship: uniform, normal, exponential, log-normal and triangular on
the continuous side; Bernoulli, binomial, Poisson, geometric and categorical
on the discrete side.

The normal, log-normal, binomial and Poisson CDFs have no closed form and are
evaluated through a shared SpecialFunctions file linked into those four
packages: incomplete gamma by series and continued fraction, incomplete beta by
continued fraction, the error function derived from the incomplete gamma rather
than approximated separately, and the normal quantile by Acklam's rational
approximation refined with one Halley step. Distributions with a small tail
declare SurvivalFunction rather than inheriting the subtraction from one, which
would round that tail away.

Tested against reference values computed outside the codebase, against the
contract each interface states, and empirically against a seeded generator.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeGiPWLApq21sJZfMyrmdH
README gains feature bullets, usage sections for both categories, and API
reference tables for IRandomProvider, IDistribution<T>, IContinuousDistribution
and IDiscreteDistribution. CLAUDE.md records the new key files, the two
provider categories, why neither interface has an async tier, and why the
small-tail distributions declare SurvivalFunction rather than inheriting the
subtraction from one. DESCRIPTION.md and TAGS.md pick up the new capabilities
for NuGet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeGiPWLApq21sJZfMyrmdH
Comment thread Shared/SpecialFunctions.cs Fixed
Comment thread Essentials.Tests/DistributionProviderTests.cs Fixed
Comment thread Essentials.Tests/DistributionProviderTests.cs Fixed
…patch]

Four findings from github-code-quality, all real, two of whose suggested
fixes would have changed behaviour for the worse.

TriangularDistributionProvider.Pdf compared a value against the mode with ==.
The check was a guard against 0/0 where the mode sits on the upper bound and
the upper side is empty, not a test of whether two computed values agreed, so
an epsilon window would have flattened the density around the peak. Branching
on the width of the empty side instead says what the code is actually doing
and drops the comparison.

SpecialFunctions.StandardNormalQuantile short circuited on a residual of
exactly zero. That case is already covered: a zero residual gives either a zero
correction, which changes nothing, or a zero times an infinity where the
density has underflowed, which the existing non-finite guard catches. The
early return is removed rather than loosened. Loosening it to a tolerance, as
suggested, would have skipped the Halley refinement for every probability
below about 1e-16, where the CDF value itself is smaller than any fixed
threshold — measured at p = 1e-20 that costs 1.3e-9 of accuracy, which is
Acklam's unrefined error. A new test pins the tail against Wichura's AS241 so
the regression cannot come back unnoticed, and covers the underflow guard and
the exactly-zero residual that the deleted branch used to handle.

The two test loops that mapped a probability to a quantile and then used only
the quantile now project the sequence with Select, as suggested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeGiPWLApq21sJZfMyrmdH
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

… [patch]

SonarCloud's quality gate failed the PR on Security Rating C for new code.
The two findings behind it are csharpsquid:S2245, "use a cryptographically
strong random number generator", on the two System.Random constructions in
NativeRandomProvider.

Suppressed with a justification rather than changed, following MD5HashProvider
and SHA1HashProvider, which carry the same shape of suppression for S4790 for
the same reason: the flagged construct is the type's entire contract, callers
select it deliberately, and it cannot be substituted for a stronger one
without the type ceasing to be what it is. CryptoRandomProvider is the
implementation of the same interface for work that has to be unpredictable,
and the remarks on both types point at each other.

Worth noting what the rule does not catch: XoshiroRandomProvider and
PcgRandomProvider are just as non-cryptographic, and go unflagged because they
spell the algorithm out rather than calling a recognised API. Deleting the
System.Random wrapper would have cleared the gate without making the library
one bit safer, so the suppression is the honest fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeGiPWLApq21sJZfMyrmdH

Copy link
Copy Markdown
Contributor Author

Analyze & Release is red on 41e8f38 because SonarCloud is down, not because of this diff.

The job died in scanner pre-processing, before it read a single source file:

Downloading from https://sonarcloud.io/api/server/version failed. Http status code is ServiceUnavailable.
An error occured while querying the server version! Please check if the server is running and if the address is correct.
Pre-processing failed. Exit code: 1

sonarcloud.io is returning 503 to everything right now — I get the same from three consecutive requests to /api/server/version outside CI. Nothing in this PR touches the scanner, the workflow, or anything it depends on, and the previous head passed this same job.

There is no fix to port: it is an upstream outage, so the only action is to re-run once the service is back. I'm holding the re-run rather than spending it into a service that is still 503, and I'm watching for recovery — I'll re-run the job and report the result.

One thing this leaves unverified. The earlier red check on 55ac40b was the SonarCloud quality gate — Security Rating C on new code, from two csharpsquid:S2245 findings on the System.Random constructions in NativeRandomProvider. 41e8f38 suppresses them with a justification, following MD5HashProvider and SHA1HashProvider, which carry the same shape of suppression for S4790 and pass the gate on main. The outage means no analysis ran against that change, so the suppression is precedent-backed but not yet confirmed. If S2245 is still reported once Sonar is back, the suppression isn't reaching the scanner and I'll take a different approach.

For what it is worth, the gate is advisory in this repository (SONAR_BLOCKING_GATE is unset), so it is the pre-processing crash rather than the gate that turned the job red.

Everything else on this head is green: .NET Workflow build and tests passed on the previous head, and 889 tests pass locally against 41e8f38 with a clean analyzer build.


Generated by Claude Code

@matt-edmondson
matt-edmondson merged commit 3d9e588 into main Sep 13, 2026
10 of 11 checks passed
@matt-edmondson
matt-edmondson deleted the claude/sharp-cannon-flnpg5 branch September 13, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants