Skip to content

coding agents / LLMs and make models fair #23

@alee

Description

@alee

Look into integrating advances in coding agents with the MMF initiative:

  1. is the narrative documentation sufficient for a coding agent / LLM to re-implement the model (tests the narrative documentation as well as coding agents). For existing code, refactoring and upgrading to modern libraries may be helpful, for no code, with customized implementation guidance for different frameworks (agents.jl, NetLogo, Mesa, RePast, Mason, cormas, GAMA, etc)
  2. come up with rubrics / prompts like agents.md as inserts into the MMF protocol to assist with these, consider including them in the candidate-model-template github repository
  3. come up with evaluation rubrics to assess whether a coding agent's reimplementation lines up with the original described model; generating similar output (pattern oriented modeling, generating similar figures within some range of error, generating equivalent data distributions, etc.) is one key part but not sufficient, the model implementation must also be in line with the narrative description so unit / integration tests would be important.
  4. this may also feed back into the ODD protocol or other documentation protocols
  5. anything I've missed?

getting this done comprehensively should result in an interesting publication...

Example model from @sgfost https://github.com/sgfost/pedestrians-with-cognitive-heuristics

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions