Hub notebook server failing to start with ExpiredAuthenticationToken

[TomAugspurger]

Currently, it seems like we aren't able to start singleuser notebook servers. After logging in and selecting a profile, the server will fail to start with a message like:

2024-05-12T23:04:09Z [Warning] AttachVolume.Attach failed for volume "pvc-{guid}" : rpc error: code = NotFound desc = Volume not found, failed with error: GET http://localhost:7788/subscriptions/{guid}/resourceGroups/MC_pcc-prod-2-rg_pcc-prod-2-cluster_westeurope/providers/Microsoft.Compute/disks/restore-{guid} -------------------------------------------------------------------------------- RESPONSE 401: 401 Unauthorized ERROR CODE: ExpiredAuthenticationToken -------------------------------------------------------------------------------- 

{
    "error": {
        "code": "ExpiredAuthenticationToken",
        "message": "The access token expiry UTC time '5/12/2024 7:52:04 PM' is earlier than current UTC time '5/12/2024 11:04:02 PM'."
    }
}

 -------------------------------------------------------------------------------

That indicates that the server failed to start because the pod can't attach the user's Persistent Volume (with the home directory). It's apparently using an expired access token to authenticate to Azure 

I'm taking a look.

[TomAugspurger]

I've rotated all of the user nodes, without any effect.

It's possible that AKS will resolve this issue itself without any intervention. I'll keep looking into this.

[TomAugspurger]

This is resolved now.

Whatever token AKS was holding onto too long as been renewed. We'll keep an eye on it but let me know if it occurs again.

[TomAugspurger]

At risk of prematurely calling this "fixed" again, I think this is resolved. Our failed pod count is down to 0 for the last 24 hours.

We'll keep monitoring it, but ping me if you see anymore of this failure.

[andrews-j]

Hi, I am unfortunately getting the same error now as I attempt to log on. I was able to log on this morning, but not now. I've changed nothing on my end.

[TomAugspurger]

Hmm thanks. I was just able to start a notebook server, so this seems to be an intermittent issue.

I've cordoned and drained the node that your pod was assigned to, in case it's a node-wide thing (though yesterday I was seeing this issue on multiple nodes).

[Websy-A]

Still not working for me and hasn't all day. I'll try again tomorrow.

[TomAugspurger]

Yep, I'm seeing it (off and on) today too. I'm following up with some teams and will post more information as I have it.