Skip to content

Add lossless DIVE KWCOCO classification support - #1851

Draft
PaulHax wants to merge 6 commits into
hierarchical-classification-pr4-lifecyclefrom
hierarchical-classification-pr5-kwcoco
Draft

Add lossless DIVE KWCOCO classification support#1851
PaulHax wants to merge 6 commits into
hierarchical-classification-pr4-lifecyclefrom
hierarchical-classification-pr5-kwcoco

Conversation

@PaulHax

@PaulHax PaulHax commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Add lossless DIVE KWCOCO classification support

Standard KWCOCO fields can store one primary category and one aligned probability vector. They
cannot preserve all DIVE confidence-pair details by themselves.

This PR defines one DIVE KWCOCO profile for Web and Desktop. A DIVE export can preserve hierarchy
edges, pair order, explicit zero values, sparse membership, and track IDs.

KWCOCO tradeoffs: why an extension field alongside standard prob

Standard KWCOCO can only express one primary category (category_id/score) plus one dense
probability vector (prob) aligned to the file's full category list. That's lossy for DIVE's
model of a track's confidence pairs:

  • Sparse vs. dense. DIVE only stores the pairs a user actually assigned. prob has no way to
    say "never assigned" vs. "assigned zero" — every category in the file gets a slot.
  • Order. DIVE's pair order is confidence-sorted, but ties break by edit history, not a
    deterministic rule (prob's order is fixed by category-list position, not edit history).
  • Hierarchy. Standard supercategory does express parent/child edges natively, so we use
    it as-is — no extension needed there.

We chose to keep writing standard category_id/score/prob (so any non-DIVE KWCOCO consumer
still gets a usable primary label) and add dive_confidence_pairs, an ordered, sparse
[[type, confidence], ...] array that round-trips DIVE's exact vector. On import, DIVE prefers
dive_confidence_pairs when present and valid; otherwise it falls back to positional-mapping
prob, then to category_id/score.

Known cost: prob is still fully dense — one float per category in the entire file, per
annotation, even though the track only has a couple of real pairs. For a large/deep hierarchy
this is a mostly-zero array repeated on every detection. Accepted for standard-field
compatibility; dive_confidence_pairs is the compact representation DIVE itself round-trips
through.

Example:

{
  "info": {
    "dive_extensions": ["dive_detection_attributes", "dive_track_attributes", "dive_notes", "dive_confidence_pairs"]
  },
  "categories": [
    { "id": 1, "name": "fish", "keypoints": ["head", "tail"] },
    { "id": 2, "name": "shark", "supercategory": "fish" },
    { "id": 3, "name": "crab" }
  ],
  "annotations": [
    {
      "id": 1,
      "image_id": 1,
      "category_id": 2,
      "score": 0.97,
      "prob": [0.03, 0.97, 0],
      "dive_confidence_pairs": [["shark", 0.97], ["fish", 0.03]],
      "track_id": 42
    }
  ]
}

category_id/score give a non-DIVE reader the primary label. prob gives a dense vector over
all 3 categories in this small example. dive_confidence_pairs is what DIVE reads back: the
track's real, ordered, 2-entry vector (shark, a child of fish, at 0.97; fish itself at 0.03).

Import

  • Map each external prob value by its category-array position.
  • Keep unnamed category positions so later values stay aligned.
  • Clamp external values to the range from 0 through 1.
  • Keep the ten highest finite external values above 0.001.
  • Prefer a valid dive_confidence_pairs extension when the file contains one.
  • Report one warning and use prob when the extension is invalid.
  • Select the vector from the annotation with the highest frame index.
  • Use the larger annotation ID to resolve a same-frame tie.
  • Use numeric filename order when an external image has no frame index.
  • Read each supercategory as a direct parent link.
  • Keep usable annotations when hierarchy data is invalid.
  • Import a missing or unnamed primary category as unknown.

For multicamera import, DIVE stores the first valid hierarchy on the parent dataset. Later cameras
must agree with it. Camera datasets do not keep derived hierarchy copies.

Export

  • Create categories for all exported pair names and all hierarchy nodes.
  • Write direct parents to supercategory.
  • Write an aligned prob array on each annotation.
  • Write the standard category_id and score compatibility fields.
  • Write ordered dive_confidence_pairs data.
  • Declare the extension in info.dive_extensions.
  • Keep the difference between a missing pair and a pair with zero confidence.
  • Apply type and threshold filters to exported copies only.
  • Validate a multicamera export before DIVE starts the ZIP response.

Manual tests

Test data: download and extract
classification.zip. Paths
below are relative to dive-classification-test-data/.

Web KWCOCO round trip

  1. Create a Web dataset from
    dive-classification-test-data/classification/single-camera-linked-types/.
  2. Export as KWCOCO, then import the export into a new dataset that uses the same images.

Verify: The hierarchy, track IDs, pair order, and confidence values survive the Web round trip.

Desktop KWCOCO round trip

  1. Create a Desktop dataset from
    dive-classification-test-data/classification/single-camera-linked-types/.
  2. Export as KWCOCO, then import the export into a new dataset that uses the same images.

Verify: The hierarchy, track IDs, pair order, and confidence values survive the Desktop round trip.

Cross-platform KWCOCO exchange

  1. Import a Web export into Desktop.
  2. Import a Desktop export into Web.

Verify: Both applications show the same hierarchy, track IDs, pair order, and confidence vectors.

Exact confidence-pair extension

  1. In Web, create an image-sequence dataset from
    dive-classification-test-data/media/image-sequence/.
  2. Import kwcoco/exact-confidence-pairs-round-trip/exact-confidence-pairs.kwcoco.json, inspect
    track 17 and the hierarchy, then export and reimport it; repeat on Desktop.

Verify: Both applications preserve sparse pair membership, stored order, and the explicit zero.

Empty confidence-pair extension

  1. In Web, create an image-sequence dataset from
    dive-classification-test-data/media/image-sequence/.
  2. Import kwcoco/empty-confidence-pairs-extension/empty-confidence-pairs.kwcoco.json, inspect the
    warning and track 7, then repeat on Desktop.

Verify: Each application reports one warning and gives track 7 the fallback vector shark: 0.8,
fish: 0.2.

Stack

This is PR 8 of 9. Previous: make lifecycle operations classification-safe.
Next: define raw and resolved boundaries.

  1. Allow devDependency imports in TypeScript spec files
  2. Preserve warnings from every Desktop import file
  3. Copy source metadata when creating a single-camera soft clone
  4. Add hierarchical track classification
  5. Centralize hierarchical classification changes
  6. Replace mutable merged tracks with read-only projections
  7. Make track lifecycle operations classification-safe
  8. Current — Add lossless DIVE KWCOCO classification support
  9. Define raw and resolved classification boundaries

This PR is stacked on hierarchical-classification-pr4-lifecycle.

Overall: 23 files, +2,020/-137. Commits: 5f2aab8f through 2593d0ec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant