Skip to content

AGENTS.md

Latest commit

 

History

History
89 lines (80 loc) · 5.94 KB

AGENTS.md

File metadata and controls

89 lines (80 loc) · 5.94 KB

AGENTS.md

This guide orients AI coding agents and developers working in this repository. It applies to the entire repo (no nested AGENTS files).

Project Snapshot

  • Purpose: cmd_queue builds and executes DAGs of shell commands via interchangeable backends (serial, tmux, slurm, experimental Airflow) with both Python and CLI frontends.
  • Entry points: cmd_queue/main.py (and python -m cmd_queue) and the cmd_queue console script defined in pyproject.toml. Rich CLI scaffolding lives in cmd_queue/cli_boilerplate.py.
  • Backbone types: queue abstractions in cmd_queue/base_queue.py with Bash job helpers in serial_queue.py, tmux_queue.py, and slurm_queue.py. The Airflow prototype is in airflow_queue.py.
  • Monitoring: monitor_app.py provides textual/rich monitors; many utilities are under cmd_queue/util/ (bash helpers, networkx graph formatting, tmux helpers, textual extensions, YAML helpers).
  • Examples & docs: Examples live in examples/. Sphinx sources are under docs/ with docs/source/index.rst as the landing page.

Repository Layout

  • cmd_queue/: core package
    • __init__.py: usage-rich module docstring, backend demos, convenience imports.
    • __main__.py: wiring for the CLI entry point.
    • base_queue.py: core queue classes and shared behaviors.
    • serial_queue.py: always-available backend producing sequential bash scripts.
    • tmux_queue.py: tmux workers with GPU pinning support; requires system tmux.
    • slurm_queue.py / slurmify.py: slurm submission logic and sbatch templating.
    • airflow_queue.py: experimental Airflow DAG generator.
    • monitor_app.py: runtime monitoring UI (rich/textual-driven).
    • util/: shared helpers (util_bash.py, util_networkx.py, util_tmux.py, util_network_text.py, util_yaml.py, richer.py, etc.) plus .pyi stubs to aid type checkers.
    • *.pyi files mirror public interfaces for static analysis (py.typed included).
  • tests/: pytest suites covering CLI flows, bash job error handling, import sanity, and tmux queue error handling (skips when tmux unavailable).
  • docs/: Sphinx configuration (Makefile, source/conf.py, source/index.rst, manual notes in source/manual/pitch.rst).
  • examples/: sample queue definitions.
  • Tooling scripts in repo root: run_developer_setup.sh, run_tests.py, run_linter.sh, run_doctests.sh, run_developer_setup.sh, publish.sh, plus packaging metadata (setup.py, pyproject.toml, MANIFEST.in).
  • dev/: helper scripts/keys for internal CI (generally not needed for day-to-day development).

Environment Setup

  • Requires Python >=3.8 (per pyproject.toml metadata). Create and activate a virtual environment before installing.
  • Quick setup (installs dependencies and editable package): 
    ./run_developer_setup.sh
  • Dependency groups:
    • Core runtime: see requirements/runtime.txt (ubelt, networkx, rich, pandas, ruamel.yaml, psutil, numpy versions keyed to Python versions).
    • Testing: requirements/tests.txt (pytest, xdoctest, pytest-cov, coverage).
    • Linting/docs/optional extras are in requirements/linting.txt, requirements/docs.txt, and requirements/optional.txt.
  • Backend system dependencies:
    • tmux backend requires tmux available on PATH.
    • slurm backend needs an active Slurm deployment (sbatch/squeue, etc.).
    • Airflow backend is experimental and assumes an existing Airflow environment.

Running & Writing Tests

  • Full suite (pytest + coverage + xdoctest): 
    python run_tests.py
    Outputs coverage HTML to htmlcov/.
  • Targeted runs: 
    pytest tests             # test modules
pytest cmd_queue         # package-level xdoctests and unit tests
./run_doctests.sh        # xdoctest all cmd_queue modules
  • CLI linting: 
    ./run_linter.sh          # flake8 on cmd_queue and tests
  • Notes:
    • Tmux-dependent tests skip automatically when tmux is missing.
    • pyproject.toml configures pytest options, doctest style, and coverage ignore rules.
    • Prefer adding doctested examples to module/function docstrings; they are executed via xdoctest.

Documentation

  • Build Sphinx docs locally: 
    make -C docs html
  • Sources are in docs/source/ with autogenerated API docs under docs/source/auto/ and narrative guides under docs/source/manual/.
  • README.rst provides a user-facing overview and quickstart examples; keep it aligned with code changes.

Working with the CLI

  • The cmd_queue CLI stores queue JSON definitions under ~/.cache/cmd_queue/cli by default (override with --dpath).
  • Common commands (see CmdQueueCLI in cmd_queue/main.py): new, submit, show, run, cleanup. cleanup kills tmux sessions prefixed with cmdq_.
  • Use --backend to select serial, tmux, or slurm. The Airflow backend emits a DAG scaffold but is not fully supported.

Extending / Refactoring Tips

  • Queue/backends: extend BaseQueue patterns in base_queue.py; mirror interfaces in corresponding .pyi stubs to keep static typing consistent.
  • Jobs: serial_queue.BashJob encapsulates command strings, logging, and dependency wiring; reuse its helpers when adding new behaviors.
  • Utilities: prefer existing helpers in cmd_queue/util/ for bash quoting, graph formatting, tmux control, and YAML serialization instead of reimplementing.
  • Monitoring: monitor_app.py integrates rich/textual views; ensure new backend events surface there for consistent UX.
  • Keep CLI and Python APIs aligned—update both docstrings and CLI boilerplate when changing flags or outputs.

Release & Packaging Notes

  • Console script is defined in pyproject.toml under [tool.xcookie.entry_points] (cmd_queue = cmd_queue.__main__:main).
  • Package metadata lives in setup.py, pyproject.toml, and MANIFEST.in; update versioning and classifiers there when publishing.
  • Continuous integration references helper scripts in the repository root; preserve compatibility when altering developer workflow commands.