Remove old code and move 0.2 README to docs, also add comprehensive API documentation

Author: OMouta

README.md

@@ -1,9 +1,57 @@
-# PyLua - Embedded Python Interpreter for Luau
+# PyLua — Embedded Python Interpreter for Luau
 
-PyLua is a Lua module that embeds a Python interpreter, allowing you to run Python code within your Luau scripts. It supports Python 3.12 syntax and below, making it lightweight and focused while still providing comprehensive Python functionality. PyLua is designed to be easy to use, making it ideal for Roblox game modding.
+> Notice: For v0.2 docs, see [docs/other/0.2.md](docs/other/0.2.md).
 
-> **⚠️ Documentation Notice**: Please read the [REWRITE_PLAN.md](internalDocs/REWRITE_PLAN.md) for details on the upcoming v0.3 rewrite. For v0.2 documentation, see [old/README.md](old/README.md).
+PyLua lets you run Python inside Luau (e.g., Roblox). The v0.3 rewrite is a proper interpreter built in Luau with a CPython-inspired design.
 
-## 📝 License
+## What is it?
 
-This repository is under the [`MIT license`](./LICENSE).
+- A Python 3.12-and-below interpreter implemented in Luau
+- Runs on [Lute] and other Luau-compatible runtimes
+- Embeddable API for executing/evaluating Python and sharing values via `globals()`
+
+## Use cases
+
+- Author gameplay logic in Python while running on Luau
+- Build modding hooks: expose Luau callbacks to Python scripts
+- Teach/prototype Python inside Roblox-like environments
+- Explore interpreter architecture (tokens → AST → bytecode → VM)
+
+## How it’s built
+
+Interpreter pipeline:
+
+- Lexer → Parser → AST → Compiler → Bytecode → VM
+
+Key modules (see `src/PyLua/`):
+
+- `lexer.luau`, `parser/` – Python-compliant tokenization and parsing
+- `compiler.luau` – compile AST to bytecode
+- `vm/` – stack-based virtual machine
+- `objects/` – Python object model
+- `builtins/` – core built-in functions and types
+
+## Status
+
+- Version: `0.3.0-dev`
+- Target: Python 3.12 syntax and below (3.13+ out of scope)
+- Roadmap: `internalDocs/REWRITE_PLAN.md`
+
+## Get started
+
+See docs and examples for usage and API details:
+
+- Docs home: `docs/README.md`
+- Examples: `docs/examples/`
+
+You can also quickly try an example with Lute from the repo root:
+
+```powershell
+lute docs/examples/hello_world.luau
+```
+
+## License
+
+MIT — see [`LICENSE`](./LICENSE).
+
+[Lute]: https://github.com/luau-lang/lute

docs/API.md

@@ -0,0 +1,178 @@
+# PyLua API
+
+This guide explains the public API users interact with when embedding Python in Luau. It reflects the current v0.3.0‑dev implementation.
+
+## Importing
+
+From this repository when running with Lute:
+
+```lua
+-- From docs/examples/* the relative import is:
+local PyLua = require("../../src/PyLua")
+
+-- From the repo root or your own project layout, adjust the path accordingly,
+-- e.g. local PyLua = require("src/PyLua").
+```
+
+## Creating a runtime
+
+```lua
+local py = PyLua.new({
+  debug = false,     -- optional; enables extra diagnostics in some subsystems
+  timeout = 0,       -- optional; planned, not enforced yet in v0.3.0‑dev
+  maxRecursion = 0,  -- optional; planned guard, not enforced yet
+  builtins = nil,    -- optional; override built-ins
+})
+```
+
+Notes:
+
+- A PyLua runtime holds its own globals and built-ins table.
+- The globals table is pre-populated with `__pylua_version__`, `__name__ = "__main__"`, and a few standard dunders.
+
+## Running code
+
+### execute(code)
+
+Run Python statements. Returns no value; exceptions raise Luau errors.
+
+```lua
+py:execute([[ 
+    x = [1, 2, 3]
+    y = { 'name': 'test', 'value': 42 }
+    print("sum:", sum(x))
+]])
+```
+
+### eval(code) -> any
+
+Evaluate a single Python expression and return its value (converted when reasonable).
+
+```lua
+local v = py:eval("1 + 2 * 3")  -- 7
+```
+
+Edge cases:
+
+- `eval` requires an expression (not statements); use `execute` otherwise.
+- Errors are thrown as Luau `error(...)`; use `pcall` to handle them.
+
+### compile(code) -> CodeObject
+
+Compile Python source to a code object (bytecode + pools).
+
+```lua
+local co = py:compile("1 + 2")
+-- co has fields: constants, names, varnames, bytecode, ...
+```
+
+### runBytecode(codeObject) -> any
+
+Execute a previously compiled code object.
+
+```lua
+local result = py:runBytecode(co)
+```
+
+## Sharing values via globals
+
+Each runtime has a separate global namespace.
+
+```lua
+-- Write from Luau, read from Python
+py:setGlobal("answer", 42)
+py:execute("print(answer)")
+
+-- Or mutate via the table directly
+local g = py:globals()
+g.threshold = 10
+py:execute("print(threshold)")
+
+-- Read back results written in Python
+py:execute("result = sum([1, 2, 3])")
+print(g.result) -- 6
+```
+
+Helpers:
+
+- `py:globals()` returns the globals table
+- `py:setGlobal(name, value)` and `py:getGlobal(name)` access single values
+
+Type notes:
+
+- Numbers, booleans, strings, lists/tuples/dicts/sets created in Python are exposed via PyLua’s object model.
+- On return from the VM, some primitives are unwrapped for convenience (e.g., `int` → Luau number).
+
+## Capturing output / customizing builtins
+
+`print(...)` is implemented in `src/PyLua/builtins/functions.luau` and supports swapping the writer for tests/tools.
+
+```lua
+local Builtins = require("src/PyLua/builtins/functions")
+local lines = {}
+Builtins.setWriter(function(s) table.insert(lines, s) end)
+
+py:execute("print('hello')")
+-- lines now contains { "hello" }
+```
+
+Providing a custom builtins table via `PyLua.new({ builtins = ... })` is also possible, but most use cases are covered by adjusting the writer.
+
+## Interop notes
+
+- Globals are the simplest way to bridge data in and out.
+- Calling Luau functions from Python is planned but not yet implemented in this build.
+  - Workaround: call from Luau by evaluating Python expressions that read/write globals.
+
+Example pattern:
+
+```lua
+-- Expose a Luau value and read back a result
+local g = py:globals()
+g.n = 5
+py:execute("result = [i*i for i in range(n)]")
+print(g.result) -- e.g., [0, 1, 4, 9, 16]
+```
+
+## Error handling
+
+Use `pcall` around API calls to catch errors raised from Python execution.
+
+```lua
+local ok, err = pcall(function()
+  py:execute("1/0")
+end)
+if not ok then
+  warn("Python error:", err)
+end
+```
+
+## Feature support and limits
+
+Target: Python 3.12 syntax and a practical core subset.
+
+Highlights supported:
+
+- Statements: assignment, if/elif/else, while/for (with break/continue), def
+- Expressions: arithmetic/bitwise, comparisons, calls, attribute/subscript
+- Collections: list/tuple/set/dict literals; list comprehensions
+- Builtins: `print`, `len`, `type`, `range`, `int/float/str/bool`, `sum/min/max`, `bytes`, `repr/ascii/format`, `isinstance`
+
+Important limitations (v0.3.0‑dev):
+
+- Chained comparisons parsed but not yet compiled
+- Dict unpacking in literals not yet compiled
+- For-loop assignment targets beyond simple `Name` unsupported
+- Function defaults/kwargs are stubbed in function creation
+- Exceptions/try-except not implemented
+- f-strings tokenized; full runtime formatting in progress
+
+See [LIMITATIONS.md](LIMITATIONS.md) for the up-to-date list.
+
+## Version
+
+The runtime exposes `__pylua_version__` in globals:
+
+```lua
+print(py:globals().__pylua_version__)
+```

docs/ARCHITECTURE.md

@@ -0,0 +1,38 @@
+# Architecture Overview
+
+This document describes the PyLua v0.3 interpreter architecture and how the pieces fit together.
+
+## Pipeline
+
+Python Source → Lexer → Parser → AST → Compiler → Bytecode → VM → Result
+
+- Lexer (`src/PyLua/lexer.luau`): Converts source text into tokens, including INDENT/DEDENT for Python’s significant whitespace.
+- Parser (`src/PyLua/parser/*`): Produces a Python-like AST from tokens (expressions, statements, function defs, loops, etc.).
+- AST (`src/PyLua/ast/*`): Node shape definitions and visitor utilities.
+- Compiler (`src/PyLua/compiler.luau`): Emits a compact Pythonic bytecode sequence from AST.
+- Bytecode (`src/PyLua/bytecode/*`): Opcodes and instruction structure used by the VM.
+- VM (`src/PyLua/vm/*`): Stack-based interpreter that executes the bytecode, managing frames and scopes.
+- Objects (`src/PyLua/objects/*`): Python object model, type registry, attribute lookup, and operators.
+- Builtins (`src/PyLua/builtins/*`): Core built-in functions (print, len, type, range, …) and small helpers.
+
+## Directory map
+
+- src/PyLua/init.luau: Public API (Runtime.new, execute, eval, compile, runBytecode, globals)
+- src/PyLua/lexer.luau: Python-compliant tokenization
+- src/PyLua/parser/: Modular parser (expressions/statements/postfix/…)
+- src/PyLua/ast/: AST node types and visitors
+- src/PyLua/compiler.luau: AST → bytecode
+- src/PyLua/bytecode/: Opcodes and instruction helpers
+- src/PyLua/vm/: Interpreter, frames, and operand stack
+- src/PyLua/objects/: Base PyObject, collections, functions, etc.
+- src/PyLua/builtins/: Built-in functions and exceptions
+
+## Design notes
+
+- CPython-inspired: names, opcodes, and phases mirror CPython where pragmatic.
+- Embeddable first: clean API boundary and globals sharing.
+- Incremental correctness: aim for Python 3.12 semantics within a practical subset; expand over time.
+
+---
+
+See also: [REWRITE_PLAN.md](../internalDocs/REWRITE_PLAN.md) for detailed milestones and checklists.

docs/AST.md

@@ -0,0 +1,14 @@
+# AST Nodes
+
+The AST mirrors Python shapes at a pragmatic subset. Nodes carry line/column info.
+
+- Base shape: `{ type: string, lineno: number, col_offset: number, ... }`
+- Expressions (subset): Constant, Name, Attribute, Subscript, BinOp, UnaryOp, Compare, Call, List, Tuple, Set, Dict, ListComp
+- Statements (subset): Expr, Assign, AugAssign, Return, If, While, For, Break, Continue, Pass, FunctionDef, Module
+
+Location fields:
+
+- `lineno`, `col_offset` are set on all nodes produced by the parser
+- Some compiler-generated helper nodes (e.g., listcomp transforms) propagate positions where feasible
+
+See `src/PyLua/ast/nodes.luau` for exact type definitions.

docs/BUILTINS.md

@@ -0,0 +1,19 @@
+# Built-in Functions
+
+Implemented in `src/PyLua/builtins/functions.luau`. Highlights:
+
+- `print(*args)` — writes space-separated values; can replace writer for testing
+- `len(x)` — length with `__len__` or type-specific rules
+- `type(x)` — returns a minimal type object: `<class 'name'>`
+- `range([start], stop[, step])` — produces an iterable `range` object
+- `int(x)`, `float(x)`, `str(x)`, `bool(x)` — conversions
+- `repr(x)`, `ascii(x)` — representation helpers
+- `format(value, spec?)` — thin wrapper over Lua string.format with Python-like coercions
+- `sum(iterable, start=0)`, `min(...)`, `max(...)`
+- `bytes(x)` — from string or list of ints
+- `isinstance(obj, type|tuple)` — simple type checks with minimal subtype map (e.g., bool ⊂ int)
+
+Notes:
+
+- `print` writer is replaceable via `Functions.setWriter` for tests
+- `bytes` returns a proper `bytes` PyObject; VM also constructs bytes from tagged constants

docs/BYTECODE.md

@@ -0,0 +1,19 @@
+# Bytecode and Instructions
+
+PyLua uses a small, Python-inspired instruction set. An instruction is `{ opcode: string, arg?: number, lineno?: number }`.
+
+Key groups:
+
+- Stack ops: `POP_TOP`, `ROT_TWO`, `ROT_THREE`, `DUP_TOP`
+- Constants and names: `LOAD_CONST`, `LOAD_NAME`, `STORE_NAME`, `LOAD_ATTR`, `STORE_ATTR`, `LOAD_SUBSCR`, `STORE_SUBSCR`
+- Binary ops: `BINARY_ADD`, `BINARY_SUBTRACT`, `BINARY_MULTIPLY`, `BINARY_DIVIDE`, `BINARY_MODULO`, `BINARY_POWER`, `BINARY_FLOOR_DIVIDE`, `BINARY_LSHIFT`, `BINARY_RSHIFT`, `BINARY_OR`, `BINARY_XOR`, `BINARY_AND`, `BINARY_MATRIX_MULTIPLY`
+- Unary ops: `UNARY_POSITIVE`, `UNARY_NEGATIVE`, `UNARY_NOT`, `UNARY_INVERT`
+- Compare: `COMPARE_OP` with arg mapping (0..9) for `<, <=, ==, !=, >, >=, is, is not, in, not in`
+- Control flow: `JUMP_FORWARD`, `POP_JUMP_IF_TRUE`, `POP_JUMP_IF_FALSE`, `JUMP_IF_TRUE_OR_POP`, `JUMP_IF_FALSE_OR_POP`, `SETUP_LOOP`, `POP_BLOCK`, `BREAK_LOOP`, `CONTINUE_LOOP`, `RETURN_VALUE`
+- Iteration: `GET_ITER`, `FOR_ITER`
+- Collections: `BUILD_LIST`, `BUILD_TUPLE`, `BUILD_SET`, `BUILD_MAP`
+- Functions: `MAKE_FUNCTION`, `CALL_FUNCTION`
+
+Constants:
+
+- Numbers, strings, bools, None; bytes literals stored as tagged constants to construct at runtime

docs/COMPILER.md

@@ -0,0 +1,47 @@
+# Compiler
+
+Compiles AST into a compact Python-like bytecode executed by the VM.
+
+- File: `src/PyLua/compiler.luau`
+- Output: CodeObject with fields: `constants`, `names`, `varnames`, `bytecode`, and metadata
+
+## Pools and indices
+
+- Constants: de-duplicated by a serialized key; nil stored as a sentinel `{ __nil = true }`
+- Names: de-duplicated; both constants and names use zero-based indices (like CPython); VM adapts to 1-based tables
+
+## Expressions
+
+- Constants → `LOAD_CONST`
+- Name (Load/Store) → `LOAD_NAME` / `STORE_NAME`
+- Attribute → `LOAD_ATTR`/`STORE_ATTR`
+- Subscript → `LOAD_SUBSCR`/`STORE_SUBSCR`
+- Binary ops → `BINARY_*` (incl. `FLOOR_DIVIDE`, `MATRIX_MULTIPLY`)
+- Unary ops → `UNARY_*`
+- Compare → `COMPARE_OP` with small int tag (0..9); currently single comparator only
+- Calls → push func then args; `CALL_FUNCTION argc`
+- Collections → `BUILD_LIST`/`BUILD_TUPLE`/`BUILD_SET`/`BUILD_MAP`
+- List comprehension → compiled into a tiny function created via `MAKE_FUNCTION` then immediately `CALL_FUNCTION`
+
+## Statements
+
+- Expr → evaluate + `POP_TOP`
+- Assign / AugAssign → load/store targets; attribute/subscript targets supported
+- If/elif/else → conditional with `POP_JUMP_IF_FALSE` and `JUMP_FORWARD` (patched)
+- While → `SETUP_LOOP` + test/jumps; supports `break`/`continue` and optional `else`
+- For → `GET_ITER` + `FOR_ITER` loop; stores to simple Name targets
+- Return → push value or `None` and `RETURN_VALUE`
+- FunctionDef → compiles nested CodeObject and creates a function via `MAKE_FUNCTION`, then `STORE_NAME`
+- Module → implicit final `return None`
+
+## Jump patching
+
+- `JUMP_FORWARD` is relative; patched to the right offset from next instruction
+- Other jumps are absolute instruction indices (converted to 1-based in VM)
+
+## Known gaps
+
+- Chained comparisons not yet emitted (parser may build them)
+- Dict unpacking in literals not yet supported
+- For-loop assignment targets beyond simple Name are not yet compiled
+- Function defaults/kwargs in `MAKE_FUNCTION` are stubbed