Deploy your own PC Hub without requesting quotas

[brunosan]

I've followed the Hub deployment and I'm having some issues. This is the script I'm using, basically copy paste from the link above.

The issue I have is that everything seems to work, but then no workers get launched. Even when there are tasks on queue, and the Logs show that workers have been requested.

If I set the min workers to 1 instead of 0, I get this deployment error

...
 "message": 
"Operation could not be completed as it results in exceeding approved LowPriorityCores quota. 
Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 3, Current Usage: 0, 
Additional Required: 6, (Minimum) New Limit Required: 6. 
Submit a request for Quota increase at https://aka.ms/[redacted] 
Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-portal/supportability/low-priority-quota"
...

Which indeed seems the case

Does this mean that, by default, I need to request more Low Priority (Spot) instances to run a Hub on these?
This also happens if I use normal cores by commenting the 3 spot lines.

"message": 
"Operation could not be completed as it results in exceeding approved Total Regional Cores quota. 
Additional details - Deployment Model: Resource Manager, Location: westeurope, Current Limit: 10, Current Usage: 8, 
Additional Required: 6, (Minimum) New Limit Required: 14. 
Submit a request for Quota increase at https://aka.ms/[redacted] by specifying parameters listed in the ‘Details’ section for deployment to succeed.
Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests"

Is there guidance to deploy a Hub without requesting quotas? Thanks!

[TomAugspurger]

Does this mean that, by default, I need to request more Low Priority (Spot) instances to run a Hub on these?

I believe that's the case. It's been a while, but IIRC there are two things you need to change two things:

1. The number of cores for that VM type
2. The "Total Regional Spot vCPUs" (https://learn.microsoft.com/en-us/azure/quotas/regional-quota-requests)

This also happens if I use normal cores by commenting the 3 spot lines.

I don't recall what all needs to change, but you might need to also update the Kubernetes taints / tolerations. For the PC Hub, I think that's at https://github.com/microsoft/planetary-computer-hub/blob/main/helm/chart/config.yaml#L306-L309 and https://github.com/microsoft/planetary-computer-hub/blob/main/helm/chart/config.yaml#L347-L352.

Finally, it might be worth double-checking whether you need a full-blown Hub deployment. If you're deploying something for multiple users it makes sense. But if you're just doing something for a single user, then you have more options.

https://docs.dask.org/en/latest/deploying-kubernetes.html gives a bit of an overview, and either Helm or Native might make sense for a single user.

Or you can bypass Kubernetes and use dask-cloudprovider to deploy directly onto VMs (https://cloudprovider.dask.org/en/latest/azure.html).