lipa: No on-chain balance after channel close

[q-src]

After a channel close on 2025-03-21 18:37:37 +01:00, the node does not detect any on-chain balance.
However, the closing transaction shows that both channel partners received funds.

Details

• Node PubKey: 021f560e5f34f258e2583082460040d4ab544bf7345f9e940e440a351fbbd25001
• Channel Close TxId: 015ab95a44df1b00bd8d0c6e2cddc5000d23e852332fed6994d91b0c79fb1123
• Channel Close Addr: bc1qhss6dgr3gsezmjw66yp7cg363zugm0kh6q9mtrhfq2hcqgjj5wrqc4ava0

Another unrelated node 030f93a711df9890d573ccc96ec3caa66924a0b8f8d67ec8f2b636b019fe98445a appears to experience the same issue.

I'm happy to share further details if needed. Thanks a lot for looking into this!

[cdecker]

Quickly updating the issue too, following our chat discussion.

I found that the node does no longer track channel 0264a62a4307d701c04a46994ce5f5323b1ca28c80c66b73c631dbcb0990d6e835-chan#2 which I believe is the one you were referring to. The following line is a bit worrisome:

0264a62a4307d701c04a46994ce5f5323b1ca28c80c66b73c631dbcb0990d6e835-chan#2: Forgetting channel: It has been 8226 blocks without the funding transaction 398b7b152e61781814e83fe8c4862a9ec03f0832150aa3307c76ba028945eaab getting deeply confirmed. We are fundee and can forget channel without loss of funds.

This is an issue we have already seen once: the its-ok-to-forget logic did not account for the fact that on zeroconf channels we may already own part of the funds. This was addressed in ElementsProject/lightning#8149, but not yet backported. I will take care of backporting this logic asap.

As for resolution of this issue, by rescanning starting before the creation of the channel we should be able to pick it up again, this time not forgetting it as it falls into the prerequisites of ElementsProject/lightning#8149.

[cdecker]

And for future reference: the session_id=7308825187057665 in which the channel was closed.

[q-src]

Thanks a lot for sharing your insights!

According to our logs, the node still tracks channel 0264a62a4307d701c04a46994ce5f5323b1ca28c80c66b73c631dbcb0990d6e835-chan#1. So I guess you are right that 0264a62a4307d701c04a46994ce5f5323b1ca28c80c66b73c631dbcb0990d6e835-chan#2 is the one which got closed.

Has the rescan already been completed? Because, as of ~10.00 AM today, the node still does not detect any on-chain funds.

[cdecker]

I'm having a bit of trouble initiating the rescan, the extra argument needed to trigger it is getting lost along the scheduling path and I need to check where.

[q-src]

It looks like another node has the same issue:

NodeState { id: “0328f5898d043b3117daa19bfe98821dec966128188de1fb89144c19edd28cd9c8”, block_height: 904734, channels_balance_msat: 0, onchain_balance_msat: 0, pending_onchain_balance_msat: 0, utxos: [], max_payable_msat: 0, max_receivable_msat: 4000000000, max_single_payment_amount_msat: 4294967000, max_chan_reserve_msats: 0, connected_peers: [], max_receivable_single_payment_amount_msat: 0, total_inbound_liquidity_msats: 0 }

The channel got closed on July 5:

{
    “alias_local”: “14325564x14155738x38222”,
    “alias_remote”: “2357895x9165199x2594”,
    “closed_at”: 1751698066,
    “closing_txid”: “31da119f4347559c0df9f1fa4b48b98a600b19e41871fc70dff17fa7ec887e44",
    “funding_outnum”: 0,
    “funding_txid”: “ef3da66f0d886c72ce1b0b2f741b145024bcc5a7a5851543caf70c0dee0c521f”,
    “htlcs”: “”,
    “local_balance_msat”: 54907000,
    “receivable_msat”: 0,
    “short_channel_id”: null,
    “spendable_msat”: 54907000,
    “state”: “Closed”
}

[cdecker]

After debugging the nodes in question we found that an old issue is the root cause:

The specification allows the fundee to forget channels that did not confirm in 2 weeks, if the fundee has no funds in the channels. These two criteria used to be identical: if the channel isn't confirmed it's not active, and we cannot have funds in them.

With the introduction and backporting of zeroconf channels, for improved user experience, we did not notice that the semantics changed, and the two criteria are no longer the same: you can now have unconfirmed, single-funded channels that the fundee has a stake in, since the channel is usable right away.

Still the forgettability was being triggered in rare cases, in which the channel really would not confirm for weeks due to a fee spike, causing the node to stop monitoring the channels in question.

We are working on a technique to recover the forgotten channel state, if at all possible, but in the meantime we are refunding users that were affected. We have also backported the fix to v24.02 and v24.11, and no new cases should appear.

[cdecker]

Ok, I have been going through nodes with desync issues, and added this one too. The good news is that the backport of CLN#8149 onto v24.02gl1 and v24.11gl1 has prevented further instances of this issue. The bad news is that since the deletion does prune data from the DB that we cannot recover, the desync tool alone is not sufficient to get this node back into working order (recover the channel at least, the rest of the node should work).

We need to refund this one, sorry 🙇