Skip to content

docs(dea): Improve Readme file to be skill compatible#489

Open
minzznguyen wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
minzznguyen:dea_readme_improvement
Open

docs(dea): Improve Readme file to be skill compatible#489
minzznguyen wants to merge 1 commit into
GoogleCloudPlatform:mainfrom
minzznguyen:dea_readme_improvement

Conversation

@minzznguyen

@minzznguyen minzznguyen commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Desciption

This PR makes changes to README.md file so that when a new user tells an AI Agent to read the README, it can start running evalbench evaluations with a small amount of guidance

For example, this can be a prompt of a user to the agent:

can you read this file -  evalbench/datasets/dea-tools/README.md
and run an eval case for me:
- User: read the schema of this public BigQuery table samples.shakespeare
- Expected Agent Result:
BigQuery Public Dataset: samples.shakespeare 
Table Schema: 
- word (STRING): A single unique word extracted from a corpus. 
- word_count (INTEGER): The number of times this word appears in the corpus. 
- corpus (STRING): The specific work (play/poem) from which the word was extracted. 
- corpus_date (INTEGER): The year in which the corpus was published.
create all new files under evalbench/sandbox

Agent generated the right YAML and dataset JSON file:

orchestrator: dea
dataset_format: dea-format
model_config: datasets/model_configs/gcp_data_engineering_agent_model.yaml
simulated_user_model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
dataset_config: sandbox/shakespeare.evalset.json
env:
  SCENARIO_ID: "dea-shakespeare-schema"
scorers:
  binary_rubric_scorer:
    model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
  dataform_cloud_compile:
    gcp_project_id: !ENV ${EVAL_GCP_PROJECT_ID}
    gcp_region: !ENV ${EVAL_GCP_PROJECT_REGION}
    timeout_seconds: 300
  dataform_cloud_run:
    gcp_project_id: !ENV ${EVAL_GCP_PROJECT_ID}
    gcp_region: !ENV ${EVAL_GCP_PROJECT_REGION}
    timeout_seconds: 300
reporting:
  csv:
    output_directory: results
  bigquery:
    gcp_project_id: !ENV ${EVAL_GCP_PROJECT_ID}
runners:
  agent_runners: 1
set_up_script: datasets/dea-tools/scripts/setup_dataform.sh
tear_down_script: datasets/dea-tools/scripts/teardown_dataform.sh
dataform_workspace_gcs_archive:
  bucket: !ENV ${EVAL_GCP_PROJECT_ID}-evalbench-archives
  path_prefix: workspaces
  gcp_project_id: !ENV ${EVAL_GCP_PROJECT_ID}
  gcp_region: !ENV ${EVAL_GCP_PROJECT_REGION}

And JSON file

{
  "scenarios": [
    {
      "id": "shakespeare-schema",
      "starting_prompt": "read the schema of this public BigQuery table samples.shakespeare",
      "max_turns": 1,
      "conversation_plan": [],
      "binary_rubric": [
        "Verify that the agent listed the table schema for samples.shakespeare with columns: word (STRING), word_count (INTEGER), corpus (STRING), corpus_date (INTEGER) and their descriptions, matching: - word (STRING): A single unique word extracted from a corpus. - word_count (INTEGER): The number of times this word appears in the corpus. - corpus (STRING): The specific work (play/poem) from which the word was extracted. - corpus_date (INTEGER): The year in which the corpus was published."
      ]
    }
  ]
}

The agent ran the correct command and returned back the result of the eval case as PASS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant