Skip to content

[TS PBT] Challenge context-scoped observed function summaries during symbolic search #403

Description

@CaelmBleidd

Implementation child of #345 and a required stage-two bounded extension. Owns helper-summary integration with the completed #355 core; it is not a core completion blocker. Builds on #397/#398 and the existing call-policy implementation; do not reopen or duplicate the TS Calls experiment in #360/#385.

Delivery stage

This is a stage-two extension of #345. Complete and record the core assessment in #405 before accepting this extension's end-to-end integration and results. Literature, interface design and focused experiments may start earlier; the extension never blocks #355 or #405.

Reuse the completed core contracts. This issue owns any extension-specific changes to observation, binding, scheduling, replay, shrinking and feedback integration, with focused follow-up PRs; do not retroactively broaden #351/#352 or require already completed core issues to reopen. A negative or inconclusive pilot is recorded honestly and informs design; it does not silently cancel this full-roadmap obligation.

Publish a separately identifiable extension configuration and evaluate it against the same original oracle with the extension disabled. Its final held-out evaluation belongs to #357 and is reported separately from the development pilot.

Goal

Test whether property-relevant observations at internal call boundaries can improve interprocedural search without promoting sampled behavior to a trusted universal contract.

Scope

  • Implement a bounded subset of deterministic pure helpers with supported scalar/collection-length relations. Record callee/source hash, call site, caller/property context, input support and observed result relation.
  • Derive summaries from original-runtime observations through [TS PBT] Infer property-relevant relational hypotheses from PBT observations #397. Preserve alias/mutation/read dependencies by restricting unsupported cases explicitly; do not model side-effecting helpers as pure.
  • Primary use: prioritize call contexts and generate counterexample goals to a summary at the real helper implementation via [TS PBT] Project observed relations and assertion goals into exact EtsIR bindings #398.
  • Evaluate speculative summary substitution only as an isolated explicit mode. It must carry assumptions into every candidate, have original-runtime replay and a budgeted no-summary search path. It cannot prove absence of bugs or justify marking paths infeasible.
  • Refine/invalidate summaries with replay counterexamples and source/context changes. Keep different call contexts separate; one context's evidence cannot constrain another without justification.
  • Reuse existing TS call models/policies and record them in configuration. Hold these policies constant across PBT-feedback comparisons so gains are not attributed to unrelated semantic-model improvements.

Definition of Done

  • At least one context-specific helper relation is observed, challenged in USVM and refined or concretely refuted.
  • A fixture detects misuse across call sites or outside observed input support; side effects and stale revisions are explicit.
  • Required challenge/prioritization mode works without trusting summaries as hard contracts.
  • [TS PBT] Evaluate causal contributions of property-aware feedback and bounded extensions #357 measures whether context-scoped summaries add to entry-only relations, including cost and misleading-summary cases. Speculative substitution may be rejected by evidence, with the negative result retained.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions