hpcperfstats-tools

Command-line tools for querying and feeding an HPCPerfStats deployment. These tools run independently of the main Django portal and talk to it via the REST API.

Installation

pip install .
# or from PyPI when published:
# pip install hpcperfstats-tools

Requires Python 3.10+.

Configuration

Tools read the API base URL from a small INI config file.

Config file: Set the HPCPERFSTATS_TOOLS_INI environment variable to the path of your INI file (e.g. the path to hpcperfstats-tools.ini.example or a copy of it).

[API]
# Base URL for the HPCPerfStats REST API
base_url = http://localhost:8000/api/

For a deployed portal, use the real base URL, e.g. https://stats.cluster.domain.edu/api/.

API key

Both tools authenticate with an API key. Keys are created in the HPCPerfStats web UI (e.g. /api-key/ or /login_prompt after signing in).

• First use: pass the key with --api-key. It will be stored in ~/.hpcperfstats-api for that API base URL.
• Later runs: the cached key is used automatically, or you can pass --api-key again.
• Tools send API keys via request headers (X-API-Key), not query-string parameters.

Tools

hpcperfstats-jobstats

Prints a job-efficiency style summary for a single Slurm job (CPU/GPU utilization, memory, etc.) using the same data as the HPCPerfStats web UI.

# Summary for job 12345 (uses config base_url and cached API key)
hpcperfstats-jobstats 12345

# Override API URL and pass key explicitly
hpcperfstats-jobstats --api-url https://stats.cluster.edu/api/ --api-key YOUR_KEY 12345

# Skip TLS verification (e.g. dev with self-signed cert)
hpcperfstats-jobstats --insecure 12345

Options:

Option	Description
jid	Job ID (required).
--api-url	Base URL for the API (default from config).
--api-key	API key (optional; otherwise uses cache).
--insecure	Disable TLS certificate verification.

hpcperfstats-sacct-gen

Runs sacct for a date range and POSTs the pipe-delimited output to the HPCPerfStats ingest endpoint. Used on a login node or host that has Slurm and network access to the portal. The API stores the data using the same logic as the portal’s sync_acct (staff API key required).

# Ingest today only (default)
hpcperfstats-sacct-gen

# Ingest a specific date range (end date exclusive)
hpcperfstats-sacct-gen 2024-01-01 2024-01-08

# Pass API key (and cache it for next time)
hpcperfstats-sacct-gen --api-key YOUR_KEY 2024-01-01 2024-01-02

Options:

Option	Description
start_date	Start date (YYYY-MM-DD or parseable). Default: today.
end_date	End date, exclusive. Default: start + 1 day.
--api-key	API key (optional; otherwise uses cache).

Requirements:

• [API] base_url must be set in your tools INI (see Configuration).
• Slurm sacct on the machine where the command runs.
• Staff-capable API key for the ingest endpoint.

Python library usage

hpcperfstats-tools also exposes reusable client helpers:

from hpcperfstats_tools import ApiClient, get_job_full_dataframe

client = ApiClient("https://stats.cluster.edu/api/", api_key="YOUR_KEY")
home = client.get_json("home/")

df = get_job_full_dataframe(
    jid="12345",
    api_url="https://stats.cluster.edu/api/",
    api_key="YOUR_KEY",
)
print(df.head())

• ApiClient centralizes auth headers, JSON calls, and redirect-safe text POST.
• get_job_full_dataframe() builds a pandas DataFrame for a job by fetching job metadata and type-detail sample series over the REST API.

License

GNU Lesser General Public License v2.1 (LGPL-2.1), same as the main HPCPerfStats package. See LICENSE.