Skip to content

Commit d8e1b00

Browse files
committed
fix: preserve filters and grouping across metric sources
Resolve complete predicates for every contributing fact before aggregation. Preserve full-join grouping keys, including NULL and composite keys, and retain outer SQL scope references. Use PostgreSQL-compatible composite equality for nullable grouping keys and add native model-to-query acceptance coverage. Reject unsupported query shapes explicitly while retaining the Metrics prototype status. Signed-off-by: tchivs <topivn@live.cn>
1 parent 3e3a6ac commit d8e1b00

7 files changed

Lines changed: 952 additions & 316 deletions

File tree

docs/concepts/metrics/overview.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ FROM __semantic.__table -- special table for simple metric queries
4141
GROUP BY ds
4242
```
4343

44-
When that model query is run, SQLMesh uses its semantic understanding of the query and metrics definitions to generate the code that is actually executed by the SQL engine:
44+
SQLMesh expands the metric into SQL equivalent to the following query (generated aliases may differ):
4545

4646
``` sql linenums="1"
4747
SELECT
@@ -60,3 +60,15 @@ FROM (
6060
```
6161

6262
SQLMesh automatically generates the correct join to use values from both the `sushi.orders` and `sushi.customers` tables.
63+
64+
## Filters and grouping across metrics
65+
66+
Queries against `__semantic.__table` treat its columns as logical dimensions. A `WHERE` predicate is resolved independently for every contributing fact source and applied before aggregation, including when the filtered dimension is not in `GROUP BY`. Derived metrics are calculated from those filtered aggregates.
67+
68+
Dimension resolution prefers a column on the fact itself. Otherwise, the column must resolve to one reachable model through the configured grains and references. An explicit dimension-table alias selects that model instead of a same-named fact column. Unknown, unreachable, or ambiguous dimensions are rejected rather than ignored.
69+
70+
Fact aggregates are combined with a full join by default. Group keys present only in a later fact are retained, and matching `NULL` group keys are combined, including composite keys. Missing metric values remain `NULL`; they are not automatically converted to zero.
71+
72+
On PostgreSQL, full joins use composite-key equality to retain `NULL` groups without the planner restriction on `IS NOT DISTINCT FROM` join conditions. Corresponding dimensions must have matching PostgreSQL types; use an explicit cast in the grouping expression when models expose different types. Metric arithmetic follows the definition's SQL dialect: use a numeric or floating-point cast for fractional ratios of integer counts, and `NULLIF(denominator, 0)` when a zero denominator should produce `NULL`.
73+
74+
The prototype rejects subqueries in metric `WHERE` filters, grouping sets, and reference paths that cannot be compiled into safe matching-key joins. This includes multi-hop paths that change reference keys. These checks do not restrict ordinary SQL scopes without `METRIC` expressions. They are not a substitute for application authorization.

docs/development.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,42 @@ Run more comprehensive tests that run on each commit:
6868
make slow-test
6969
```
7070

71+
### PostgreSQL Metrics acceptance
72+
73+
The native Metrics suite loads a temporary SQLMesh project, applies its models to PostgreSQL, compiles queries with `Context.rewrite`, and checks results returned by PostgreSQL. It uses the existing `inttest_postgres` gateway and `postgres` / `docker` test markers. Only scheduling metadata uses an isolated in-memory DuckDB connection; model and metric SQL run on PostgreSQL. Temporary model schemas are cleaned up by the integration fixtures.
74+
75+
With the project's PostgreSQL test service available (see `make engine-postgres-up`), run:
76+
77+
```bash
78+
pytest tests/core/engine_adapter/integration/test_integration_metrics.py -q
79+
```
80+
81+
On a Linux Docker host, a separate test container can instead be run without publishing a database port. After activating the development virtual environment:
82+
83+
```bash
84+
(
85+
set -eu
86+
name="sqlmesh-metrics-pg-$$"
87+
docker network create --internal "$name"
88+
trap 'docker rm -f "$name" >/dev/null 2>&1 || true; docker network rm "$name" >/dev/null 2>&1 || true' EXIT
89+
docker run -d --rm --name "$name" --network "$name" \
90+
-e POSTGRES_HOST_AUTH_METHOD=trust postgres:16-alpine
91+
ready=false
92+
for attempt in $(seq 1 30); do
93+
if docker exec "$name" pg_isready -h 127.0.0.1 -U postgres; then
94+
ready=true
95+
break
96+
fi
97+
sleep 1
98+
done
99+
"$ready"
100+
export DOCKER_HOSTNAME="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$name")"
101+
pytest tests/core/engine_adapter/integration/test_integration_metrics.py -q
102+
)
103+
```
104+
105+
The trust-authenticated container is for disposable tests only: its internal network has no published host port or persistent data volume. Do not point this suite at a business database. Personal SQLMesh gateway overrides must not redirect `inttest_postgres` away from the intended test instance.
106+
71107
## Documentation
72108

73109
In order to run the documentation server, you will need to install the dependencies by running the following command.

0 commit comments

Comments
 (0)