Add new metrics for long running transactions

[joaofoltran]

Updated both metrics for long running transactions. One returns 4 metrics (one for each threshold, 1min, 5min, 10min, 30min) and another one returns the duration of the longest running transaction.

With this we can check if there was a long running transaction during the time of any issues that could cause xmin/lsn retention.

The original code was just counting how many transactions and then bringing the longest running one. This new code we know if there are multiple long running ones.

Next I'll be adding these to our grafana metrics so we can check them when diagnosing issues.

[mble]

med: I don't love this. We can re-express all this in a single query:

SELECT
  count(*) FILTER (
    WHERE
      extract('epoch', clock_timestamp() - xact_start) >= 60
  )
    AS count_60,
  count(*) FILTER (
    WHERE
      extract('epoch', clock_timestamp() - xact_start)
      >= 300
  )
    AS count_300,
  count(*) FILTER (
    WHERE
      extract('epoch', clock_timestamp() - xact_start)
      >= 600
  )
    AS count_600,
  count(*) FILTER (
    WHERE
      extract('epoch', clock_timestamp() - xact_start)
      >= 1800
  )
    AS count_1800,
  COALESCE(
    max(extract('epoch', clock_timestamp() - xact_start)),
    0
  )
    AS oldest_timestamp_seconds
FROM
  pg_catalog.pg_stat_activity
WHERE
  state IS DISTINCT FROM 'idle'
  AND query NOT LIKE 'autovacuum:%'
  AND xact_start IS NOT NULL;

Can then break it up based out of the results.

[joaofoltran]

oh, yes, much better, will update :)

[mble]

med: With the updated query, we can use QueryRowContext instead to avoid manual row closure handling.

[joaofoltran]

Updated applying your changes @mble , check it out see if it improved :)
(has been quite some time since I've touched golang)

[mble]

note: breaking change as this goes from pg_long_running_transactions to pg_long_running_transactions_count.

[joaofoltran]

We are not using these metrics anywhere, so it shouldn't break any dashboards. And I dont intend to push this to the upstream. The metric inherently changed, we have a count for each threshold, so its a different "metric" warranting a different name, wdyt?

[mble]

Was mostly just a call out that if there were consumers, it was a breaking change. No other action needed.

[mble]

opt: Very minor optimisation here is to extract this into a CTE to avoid calculating the whole EXTRACT(EPOCH FROM clock_timestamp() - pg_stat_activity.xact_start 5 times.

[joaofoltran]

yep! for sure much better

[mble]

We don't actually have tests running in CI 😬 but these changes will break the relevant test in https://github.com/planetscale/postgres_exporter/blob/main/collector/pg_long_running_transactions_test.go.

[joaofoltran]

The change in the metric itself will break the tests, so I'll update them too.

[mble]

Very mild thing, but a loop here would be a little cleaner:

  thresholds := []string{"60", "300", "600", "1800"}
  counts := []float64{count60s, count300s, count600s, count1800s}

  for i, threshold := range thresholds {
      ch <- prometheus.MustNewConstMetric(
          longRunningTransactionsCount,
          prometheus.GaugeValue,
          counts[i],
          threshold,
      )
  }

[mble]

@joaofoltran I've re-added CI so if you could rebase, that would be great.