-
Notifications
You must be signed in to change notification settings - Fork 101
Document deleting orphaned Kubernetes objects #3413
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
f2d63b6
bc8251e
16879cc
382dcda
da45408
d5d6c08
452e9db
ce4cb01
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| --- | ||
| layout: src/layouts/Default.astro | ||
| pubDate: 2026-08-19 | ||
| modDate: 2026-08-21 | ||
| navSection: Deleting orphaned objects | ||
| title: Deleting orphaned objects | ||
| navTitle: Deleting orphaned objects | ||
| description: How to review and delete Kubernetes objects that Octopus no longer deploys but that are still running in your cluster. | ||
| navOrder: 60 | ||
| --- | ||
|
|
||
| When a deployment stops including an object that a previous release deployed, Octopus marks that object as **Orphaned** in Kubernetes Live Object Status, and it keeps running in your cluster until someone removes it. Octopus can delete orphaned objects for you, one at a time or in bulk, from the project's Live Status page. | ||
|
|
||
| ## Requirements | ||
|
|
||
| To delete orphaned objects you need: | ||
|
|
||
| - Octopus Server 2026.3 or later | ||
| - Every Kubernetes monitor in the application on agent version 2.38.3 or later (v2) / 3.0.1 or later (v3), the same requirement as orphan tracking itself | ||
| - The `DeployedResourceAdminister` permission for the project, environment, and tenant you are deleting in | ||
| - A Kubernetes agent whose service account is allowed to delete the objects in question | ||
| - Objects reported by the [Kubernetes monitor](/docs/kubernetes/targets/kubernetes-agent/kubernetes-monitor). Objects observed through Argo CD are never orphaned and cannot be deleted this way | ||
|
|
||
| ## Delete orphaned objects | ||
|
|
||
| Open your project and select **Live Status** for the environment (and tenant) you want. The table offers a delete action on every orphaned row, and when an application has orphaned objects the page also shows a card counting them. To list only orphaned objects, use the **Sync status** filter and select **Orphaned**. | ||
|
|
||
| :::figure | ||
|  | ||
| ::: | ||
|
|
||
| ### A single object | ||
|
|
||
| 1. Open the row's **Resource actions** menu and choose **Review & delete**. | ||
| 2. Confirm in the **Delete orphaned resource** dialog by choosing **Delete**. | ||
|
|
||
| You can also select the object to open its detail drawer and use the **Delete** button there. | ||
|
|
||
| ### Several objects at once | ||
|
|
||
| 1. Choose **Review & delete** on the orphaned-objects card to open the **Delete orphaned resources** drawer. | ||
| 2. Select the objects to delete. Objects are grouped by deployment target, so you can select a whole target or everything at once. | ||
| 3. Choose **Next**. | ||
| 4. Review the **Confirm deletion** step, which lists every selected object grouped by deployment target. | ||
| 5. Choose **Delete**. | ||
|
|
||
| ## The deletion task | ||
|
|
||
| Each request queues one **Delete Kubernetes resources** task, which is linked from the Kubernetes resource being deleted. Deletion tasks can run at the same time as deployments. | ||
|
|
||
| The deletion task runs a script on the Kubernetes Agent that calls `kubectl delete` with a 5 minute timeout for each resource. If some objects delete and others don't, the task deletes everything it can, records every outcome, and then fails. | ||
|
|
||
| Octopus validates your selection when you confirm, and silently skips an object when: | ||
|
|
||
| - The object is no longer orphaned, most likely because a deployment re-added it. | ||
| - A deletion task for the object is already queued or running. | ||
| - The monitor no longer reports the object in the cluster, so there is nothing to delete. | ||
| - Status information for the object is stale, so Octopus cannot safely delete it. | ||
|
|
||
| ## Permissions | ||
|
|
||
| Deleting an orphaned object needs permission in two places: in Octopus, to request the deletion, and in your cluster, for the agent that carries it out. | ||
|
|
||
| ### Octopus permissions | ||
|
|
||
| Requesting a deletion requires the `DeployedResourceAdminister` permission ("Administer deployed resources managed by Octopus"), scoped to the project, environment, and tenant. It is included by default in the **Project Contributor**, **Project Deployer**, **Project Lead**, and **Space Manager** built-in roles, and is added to those roles on existing instances when you upgrade. | ||
|
|
||
| ### Cluster permissions | ||
|
|
||
| Octopus deletes objects by running `kubectl delete` on the Kubernetes agent, so the deletion uses the **agent's** service account. Make sure that account is allowed to delete the kinds and namespaces you expect to clean up. When you use the [permissions controller](/docs/kubernetes/targets/kubernetes-agent/granular-permissions) to scope permissions per step, deletions run under the reserved step slug `octopus-kubernetes-resource-deletion`. | ||
|
|
||
| ## Auditing | ||
|
|
||
| Every deletion task writes one audit event to the **Audit** screen in **Configuration**, succeeded or failed, recorded against the user who requested it. Its details list every object with its outcome, grouped by deployment target, with failures first. | ||
|
|
||
| ## Known limitations | ||
|
|
||
| - Only top-level orphaned objects can be deleted. Child objects, such as Pods belonging to a Deployment can't be deleted from Octopus to restart a service. | ||
| - You cannot stop tracking an orphaned object without deleting it. If the object is still needed by something else, delete it and recreate it from a different project. | ||
|
|
||
| ## Learn more | ||
|
|
||
| - [Kubernetes Live Object Status](/docs/kubernetes/live-object-status) | ||
| - [Orphaned objects](/docs/kubernetes/live-object-status#orphaned-objects) | ||
| - [Troubleshooting Live Object Status](/docs/kubernetes/live-object-status/troubleshooting) | ||
| - [Kubernetes agent permissions controller](/docs/kubernetes/targets/kubernetes-agent/granular-permissions) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| --- | ||
| layout: src/layouts/Default.astro | ||
| pubDate: 2025-03-28 | ||
| modDate: 2026-05-24 | ||
| modDate: 2026-08-21 | ||
| navSection: Live Object Status | ||
| title: Kubernetes Live Object Status | ||
| navTitle: Overview | ||
|
|
@@ -85,24 +85,27 @@ Sync Status tracks whether the changes Octopus deployed still matches the resour | |
| | Out of Sync | <i class="fa-solid fa-arrow-up orange"></i> | Object manifest is not the same as what was applied | | ||
| | Unknown | <i class="fa-solid fa-question grey"></i> | We don't have information about the live status of this object | | ||
| | Orphaned | <i class="fa-solid fa-link-slash orange"></i> | Object was deployed in a previous release but is no longer part of the latest deployment process | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we wanted to make this icon's color match the portal you can add |
||
| | Deleting | <i class="fa-solid fa-hourglass blue"></i> | Octopus is deleting this orphaned object. Select the status to open the task doing the work | | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Slightly off - the UI shows "Deleting" but the sync status doesn't actually change over the wire |
||
|
|
||
| Take a look at our [troubleshooting guide](/docs/kubernetes/live-object-status/troubleshooting) for details on why you may see some object statuses | ||
|
|
||
| ### Orphaned objects | ||
|
|
||
| When you deploy a project that no longer includes a resource that was deployed in a previous release (for example, you remove a resource from a YAML manifest or remove a step from the deployment process), Octopus marks the dropped resource as **Orphaned** in the Live Status table. | ||
| When a deployment succeeds and no longer includes an object that was deployed in a previous release (for example, you remove an object from a YAML manifest or remove a step from the deployment process), Octopus marks the dropped object as **Orphaned** in the Live Status table. | ||
|
|
||
| Orphaned objects: | ||
|
|
||
| - keep their existing Health Status, so Octopus continues tracking them while they remain in your cluster | ||
| - have **Orphaned** as their Sync Status (shown with the link-slash icon in the table above) | ||
| - are summarized by a total-orphans count shown on the project's Live Status page | ||
| - can be narrowed to by using the Live Status table's filter and selecting **Orphaned** | ||
| - have **Orphaned** as their Sync Status | ||
| - are excluded from the rolled-up Application Health Status and Application Sync Status | ||
| - can be deleted directly from Octopus, see [deleting orphaned objects](/docs/kubernetes/live-object-status/deleting-orphaned-objects) | ||
|
|
||
| The orphan state clears automatically on the next deployment that re-adds the resource. If you remove an orphaned object from your cluster directly (for example with `kubectl delete`), the Kubernetes monitor detects the removal and Octopus stops tracking the resource, so the orphan entry disappears from the Live Status table without any manual intervention. | ||
| Orphans are detected only after a **successful** deployment, so a failed deployment never marks anything as orphaned. | ||
|
|
||
| Skipping a step, or disabling it, does not orphan the objects that step deploys. Octopus only orphans an object when the step that deployed it ran and no longer produces it, or when that step has been removed from the deployment process altogether. | ||
|
|
||
| :::div{.info} | ||
| Orphaned-resource tracking requires every Kubernetes monitor in the application instance to be on agent version 2.38.3 or later (v2) / 3.0.1 or later (v3). On clusters with any older agent, the resource is silently removed from the Live Status table when it is dropped from a deployment, matching the previous behavior. | ||
| Orphaned-object tracking requires every Kubernetes monitor in the application instance to be on agent version 2.38.3 or later (v2) / 3.0.1 or later (v3). On clusters with any older agent, the object is silently removed from the Live Status table when it is dropped from a deployment, matching the previous behavior. | ||
| ::: | ||
|
|
||
| ### Detailed object information | ||
|
|
@@ -215,6 +218,8 @@ Viewing the data returned from the Kubernetes monitor from within Octopus requir | |
|
|
||
| This data includes the resource and application status, as well as pod logs and events for objects being monitored. This may be a change in security posture that your team should carefully consider. | ||
|
|
||
| Deleting an orphaned object additionally requires the `DeployedResourceAdminister` permission, and the Kubernetes agent's own service account must be allowed to delete the object. See [deleting orphaned objects](/docs/kubernetes/live-object-status/deleting-orphaned-objects#permissions). | ||
|
|
||
| ## Secrets | ||
|
|
||
| ### Octopus sensitive variables | ||
|
|
@@ -256,11 +261,11 @@ This setting defaults to on for all projects, but may change in the future. | |
|
|
||
| ## Known issues and limitations | ||
|
|
||
| ### Excluded steps | ||
| ### Excluded steps on older agents | ||
|
|
||
| The desired object list is compiled from objects that were applied during the last deployment. If steps are excluded during a deployment, then live status will not be shown for objects that were applied in those steps. | ||
| Where every Kubernetes monitor in the application is on agent version 2.38.3 or later (v2) / 3.0.1 or later (v3), Octopus keeps showing live status for objects deployed by a step that was skipped or disabled in the latest deployment. | ||
|
|
||
| Please avoid skipping steps that deploy Kubernetes objects. | ||
| On clusters with any older agent, objects applied by excluded steps are still removed from the Live Status table, so avoid skipping steps that deploy Kubernetes objects until every agent is upgraded. | ||
|
|
||
| ### Runbooks are not supported | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,7 +1,7 @@ | ||||||
| --- | ||||||
| layout: src/layouts/Default.astro | ||||||
| pubDate: 2025-03-28 | ||||||
| modDate: 2026-01-21 | ||||||
| modDate: 2026-08-21 | ||||||
| navTitle: Troubleshooting | ||||||
| title: Troubleshooting | ||||||
| navSection: Troubleshooting | ||||||
|
|
@@ -86,3 +86,29 @@ If possible, we recommend ensuring that | |||||
|
|
||||||
| - Octopus is the only entity to modify your deployments | ||||||
| - You craft your Kubernetes manifests to ensure that there are no invalid fields | ||||||
|
|
||||||
| ### An object stays orphaned after I re-added the step \{#orphan-persists-after-re-adding-a-step} | ||||||
|
|
||||||
| Octopus clears the orphan state when a deployment deploys the object again. Re-adding a step that is disabled does not deploy anything, so the object stays orphaned. Enable the step and deploy, or [delete the orphaned object](/docs/kubernetes/live-object-status/deleting-orphaned-objects) if you no longer want it. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I think this is a little bit confusing, it kinda conflicts with what is mentioned below
I think it needs to be explicit about re-adding a step that was removed from the process
Suggested change
|
||||||
|
|
||||||
| An object is marked [Orphaned](/docs/kubernetes/live-object-status#orphaned-objects) when a deployment succeeds and no longer produces an object that a previous release deployed. The usual causes are removing the object from a manifest, or removing the step that deployed it from the deployment process. Skipping or disabling a step does not orphan its objects, and a failed deployment never orphans anything. | ||||||
|
|
||||||
| If you expected an object to be orphaned and it isn't, check that every Kubernetes monitor in the application is on agent version 2.38.3 or later (v2) / 3.0.1 or later (v3). Orphan tracking is all-or-nothing across the application, so one older agent disables it everywhere and dropped objects are silently removed from the table instead. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. By application do you mean Octopus Deploy itself or an application instance? Seems a bit weird if one outdated agent that might be in a completely different space could disable orphan tracking for the entire Octopus instance
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. monitors deployed to from the P/E/T - so application instance |
||||||
|
|
||||||
| ## Deleting orphaned objects \{#deleting-orphaned-objects} | ||||||
|
|
||||||
| ### Octopus says an object was deleted but it's still in my cluster \{#deleted-object-still-present} | ||||||
|
|
||||||
| Deleting an object deletes its dependents in the background, so a successful deletion returns as soon as the object itself is gone. Kubernetes garbage collects what it owned afterwards, which means deleting a Deployment reports success while its ReplicaSets and Pods are still being removed. | ||||||
|
|
||||||
| ### A deletion failed \{#deletion-failed} | ||||||
|
|
||||||
| Open the deletion task from the warning icon beside the object. The task log contains detailed error messages to help troubleshooting. | ||||||
|
|
||||||
| The most common cause is cluster permissions. Deletions run on the Kubernetes agent using the agent's service account, so the agent needs `delete` on that kind and namespace. See [cluster permissions](/docs/kubernetes/live-object-status/deleting-orphaned-objects#cluster-permissions). | ||||||
|
|
||||||
| ### I can't delete an object because its status is stale \{#cannot-delete-stale-object} | ||||||
|
|
||||||
| Octopus refuses to delete an object whose status information is stale, because it can't confirm what is actually in the cluster. Status goes stale after 10 minutes without an update from the Kubernetes monitor. | ||||||
|
|
||||||
| Follow [failed to establish connection with Kubernetes monitor](/docs/kubernetes/live-object-status/troubleshooting#failed-to-establish-connection-with-kubernetes-monitor) to get the monitor reporting again, then retry the deletion. | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need a screenshot without names etc.