From 61a5b05eaa745113959a9322fb92e34c01918b28 Mon Sep 17 00:00:00 2001 From: AI Dev Date: Tue, 25 Aug 2026 14:55:31 +0000 Subject: [PATCH 1/2] Give the suite somewhere to run that is not the deploy host The Gemfile pins Ruby 3.1.7 and the app needs MariaDB, libvips and libheif. Almost no development machine has that combination, so verification has been happening in CI or on the deploy host -- which means the loop between writing a change and finding out it fails is a push and a wait. This adds a dev/test stack: compose.yaml plus docker/Dockerfile.dev, deliberately separate from the root Dockerfile, which is the production build and should not grow development concerns. The image carries only the OS libraries the gems bind to; the tree is bind-mounted and gems live in a volume, so a source edit never rebuilds anything. Its build context is ./docker rather than the repository root, so the daemon is not handed node_modules on every build. Three things it has to get right, all of them found by running it: - libvips and libheif come from the OS packages, matching the production runtime stage. The 'vips' gem is gone from the Gemfile and must stay gone. - Yarn 4, via corepack, because yarn.lock carries the Berry __metadata header that a v1 yarn cannot read. - RUBYOPT=--disable-error_highlight. Ruby 3.1 activates its own bundled error_highlight 0.3.0 before Bundler runs, and the development group asks for >= 0.4.0, so bin/rails dies in config/boot.rb with "You have already activated error_highlight 0.3.0". Neither the production image nor CI hits this because both exclude the development group; this container is the only place that installs it, and it has to, since rubocop and i18n-tasks are there. Upgrading RubyGems does not help -- gem update --system 3.4.22 was tried and the conflict is unchanged. rubocop-performance joins the Gemfile. .rubocop.yml has had `require: rubocop-performance` all along, so rubocop could not start at all without it; that went unnoticed for the same reason the rest of this did. development.rb gets two settings that only ever mattered off-localhost. LAN hosts, so the dev server can be opened from a phone -- written /.*\.local(:\d+)?\z/ because HostAuthorization matches a Regexp against the Host header including the port, while Strings and IPAddrs are matched port-stripped, so the obvious spelling 403s every request to a host:3010. And i18n.fallbacks = true, which production already sets: without it development is the only environment that renders "translation missing", so a gap in ru or zh looks broken here and invisible there. Verified in the container: 231 runs, 873 assertions, 0 failures. rubocop runs (742 offences over 111 files, the pre-existing baseline); development.rb goes from 4 offences to 3, having lost a stray blank line next to the new stanza. i18n-tasks reports 0 missing and 102 unused. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015hvzXBErEjEGMyRme2K3hi --- CLAUDE.md | 6 ++- Gemfile | 3 ++ Gemfile.lock | 4 ++ compose.yaml | 71 ++++++++++++++++++++++++++++ config/environments/development.rb | 17 +++++++ docker/Dockerfile.dev | 74 ++++++++++++++++++++++++++++++ docker/db-init.sql | 12 +++++ 7 files changed, 186 insertions(+), 1 deletion(-) create mode 100644 compose.yaml create mode 100644 docker/Dockerfile.dev create mode 100644 docker/db-init.sql diff --git a/CLAUDE.md b/CLAUDE.md index f6b055c..82a59b9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,10 +10,14 @@ The OpenIPC project website — a Rails 7.0 app (Ruby 3.1.2, MySQL) that serves - `bin/dev` — start the full dev stack via foreman (`Procfile.dev`): Rails server on **port 3010** (not 3000), `yarn build --watch` (esbuild JS), and `yarn watch:css` (sass→postcss). Use this, not `bin/rails server` alone, or assets won't rebuild. - `bin/setup` — idempotent dev bootstrap (`bundle`, `db:prepare`, clear logs/tmp, restart). +- `docker compose run --rm web ` — run anything against Ruby 3.1.7 + MariaDB without + installing either. `compose.yaml` + `docker/Dockerfile.dev` are the dev/test stack; the + root `Dockerfile` is the unrelated production build. Use this when the host Ruby does not + match `.ruby-version` — which is most hosts. Note `bundle exec rubocop`, not bare `rubocop`. - `bin/rails test` — run tests (Minitest, parallelized across cores, fixtures auto-loaded). The MySQL `test` DB is regenerated from `development`. - `bin/rails test test/models/admin_test.rb` — single file; append `:LINE` to run one test. - `bin/rails test:system` — Capybara + selenium system tests. -- `rubocop` — lint (config in `.rubocop.yml`: `rubocop-performance`, line length 120). +- `rubocop` — lint (config in `.rubocop.yml`: `rubocop-performance`, line length 120). Baseline on master is 742 offences over 111 files; judge a change by whether it adds any to the files it touches, not by the total. - `i18n-tasks missing` / `i18n-tasks unused` — audit translations (config in `config/i18n-tasks.yml`); `easy_translate` provides machine translation via `GOOGLE_TRANSLATE_API_KEY`/`DEEPL_TRANSLATE_API_KEY`. - `tools/webui-gallery/run.sh --camera ` — rebuild the WebUI screenshots on `/web-interface` from a real camera. Needs Docker and network access to the camera; everything else is in the image it builds. Run it when the WebUI changes shape (every few months). It redacts the camera's identity, substitutes a scene over the live player, refuses to open the CGIs that reset or reboot on render, and fails the run rather than installing if anything identifying survives. `tools/webui-gallery/README.md` has the traps. - Asset bundling (normally run by `bin/dev`): `yarn build` (JS → `app/assets/builds/`), `yarn build:css` (sass + autoprefixer). `app/assets/builds/` is gitignored — rebuild after JS/SCSS changes. diff --git a/Gemfile b/Gemfile index 27d8231..8d81611 100644 --- a/Gemfile +++ b/Gemfile @@ -68,6 +68,9 @@ group :development do gem 'easy_translate', '~> 0.5.1' gem 'i18n-tasks' gem 'rubocop' + # .rubocop.yml has `require: rubocop-performance`, so rubocop cannot start + # without this -- it was required by the config but never listed here. + gem 'rubocop-performance' # gem 'rubocop-rails' # gem 'ruby-debug-ide' end diff --git a/Gemfile.lock b/Gemfile.lock index d6deb8e..a9dec84 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -258,6 +258,9 @@ GEM unicode-display_width (>= 2.4.0, < 3.0) rubocop-ast (1.30.0) parser (>= 3.2.1.0) + rubocop-performance (1.20.2) + rubocop (>= 1.48.1, < 2.0) + rubocop-ast (>= 1.30.0, < 2.0) ruby-progressbar (1.13.0) ruby-vips (2.2.0) ffi (~> 1.12) @@ -338,6 +341,7 @@ DEPENDENCIES puma rails (~> 7.0.8) rubocop + rubocop-performance sassc-rails selenium-webdriver sprockets-rails diff --git a/compose.yaml b/compose.yaml new file mode 100644 index 0000000..73d365e --- /dev/null +++ b/compose.yaml @@ -0,0 +1,71 @@ +# Local development and test stack. +# +# This is the stack you develop against; it is not how the site is deployed. +# Production runs the root Dockerfile as a container behind the host's nginx -- +# see deploy/docker-compose.yml and deploy/DEV-VALIDATION.md. +# +# docker compose run --rm web bundle install # first time, and after Gemfile changes +# docker compose run --rm web yarn install --immutable +# docker compose run --rm web bin/rails db:prepare +# docker compose run --rm web bin/rails test +# docker compose run --rm web bundle exec rubocop +# docker compose run --rm web bundle exec i18n-tasks missing +# +# `bundle exec` for the development-group tools: BUNDLE_BIN puts binstubs on +# PATH, but bundler only writes one for a gem that ships an executable it knows +# about at install time, and rubocop's is not there. +# +# `docker compose up web` serves http://localhost:3010, but the JS and CSS +# bundles are built separately -- app/assets/builds/ is gitignored, and +# Sprockets raises on any page that goes through the layout until they exist: +# +# docker compose run --rm web yarn build +# docker compose run --rm web yarn build:css +services: + db: + # Matches the MariaDB that CI runs the suite against. 11.8 defaults utf8mb4 + # to uca1400_ai_ci rather than general_ci, which is why config/database.yml + # pins the collation rather than inheriting it. + image: mariadb:11.8 + environment: + MARIADB_ROOT_PASSWORD: root + MARIADB_USER: www + MARIADB_PASSWORD: www + MARIADB_DATABASE: openipc_development + volumes: + - db-data:/var/lib/mysql + - ./docker/db-init.sql:/docker-entrypoint-initdb.d/90-grants.sql:ro + healthcheck: + test: ['CMD', 'healthcheck.sh', '--connect', '--innodb_initialized'] + interval: 3s + timeout: 5s + retries: 20 + + web: + build: + # ./docker, not the repository root: the image copies nothing from the + # tree, and a root context would ship node_modules to the daemon. + context: ./docker + dockerfile: Dockerfile.dev + command: bash -c 'bundle check || bundle install; exec bin/rails server -b 0.0.0.0 -p 3010' + environment: + # Setting this is what makes config/database.yml connect over TCP instead + # of the /run/mysqld socket, which does not exist in this container. + OPENIPC_DATABASE_HOST: db + OPENIPC_DATABASE_PORT: 3306 + ports: + - '3010:3010' + volumes: + - .:/app + - bundle:/bundle + # node_modules is Linux-native and must not be shadowed by, or written + # back to, whatever the host happens to have installed. + - node-modules:/app/node_modules + depends_on: + db: + condition: service_healthy + +volumes: + bundle: + db-data: + node-modules: diff --git a/config/environments/development.rb b/config/environments/development.rb index 537e8b9..e7ff1e9 100644 --- a/config/environments/development.rb +++ b/config/environments/development.rb @@ -19,6 +19,17 @@ config.hosts << "openipc.org" + # Reach the dev server from other devices on the LAN -- a phone, or a second + # machine -- rather than only from localhost. + # + # Rails 7 already ships IPAddr 0.0.0.0/0 in the development defaults, so bare + # IP literals are allowed out of the box; only NAMED hosts need listing. + # + # The port suffix in the regex is not decoration. HostAuthorization matches + # String and IPAddr entries against the Host header with the port stripped, + # but Regexp entries against the header INCLUDING it. Written as + # /.*\.local\z/ this rule would 403 every request to trainer-arch.local:3010. + config.hosts << /.*\.local(:\d+)?\z/ # Enable/disable caching. By default caching is disabled. # Run rails dev:cache to toggle caching. @@ -62,6 +73,12 @@ # Suppress logger output for asset requests. config.assets.quiet = true + # Behave like production: a key missing from ru or zh falls back to the + # English text. Without this, development is the only environment that renders + # a "translation missing" span, so a gap looks broken here and invisible there + # -- or worse, the reverse, and a gap gets shipped because it looked fine. + config.i18n.fallbacks = true + # Raises error for missing translations. # config.i18n.raise_on_missing_translations = true diff --git a/docker/Dockerfile.dev b/docker/Dockerfile.dev new file mode 100644 index 0000000..9a88b4b --- /dev/null +++ b/docker/Dockerfile.dev @@ -0,0 +1,74 @@ +# Development and test image for the openipc.org Rails app. +# +# This is NOT the image that serves traffic -- that is the Dockerfile at the +# repository root, which is a two-stage production build. This one exists so the +# suite, rubocop and i18n-tasks can be run on a machine whose system Ruby is not +# 3.1.7, which is every machine we develop on. +# +# It carries only the OS libraries the gem set binds to. The application itself +# is bind-mounted by compose.yaml and gems install at runtime into a persistent +# volume, so a source change never rebuilds this image. +# +# The build context is ./docker, not the repository root: nothing here copies +# the application in, and a root context would ship the whole tree (node_modules +# and all) to the daemon on every build. + +# Matches the production image and .ruby-version. Do not drift: a gem that +# compiles here has to compile there. +FROM ruby:3.1.7-slim-bookworm + +ARG NODE_MAJOR=20 + +# build-essential is not optional: sassc compiles libsass from source and +# mysql2 needs the libmysqlclient headers. +# +# libvips42 and libheif1 come from the OS, not from the 'vips' gem -- the gem +# ships its own prebuilt libvips and shadowed the system copy. Snapshot's HEIF +# uploads only decode if libvips was built against libheif, which the Debian +# package is. +RUN apt-get update -qq && apt-get install --no-install-recommends -y \ + build-essential \ + ca-certificates \ + curl \ + default-libmysqlclient-dev \ + default-mysql-client \ + git \ + gnupg \ + libffi-dev \ + libheif1 \ + libssl-dev \ + libvips42 \ + libyaml-dev \ + pkg-config \ + zlib1g-dev \ + && curl -fsSL https://deb.nodesource.com/setup_${NODE_MAJOR}.x | bash - \ + && apt-get install --no-install-recommends -y nodejs \ + && rm -rf /var/lib/apt/lists/* + +# The project is on Yarn 4 (Berry); yarn.lock carries the __metadata header and +# a v1 yarn cannot read it. corepack resolves the exact version from the +# "packageManager" field in package.json, the same way the production image does. +ENV COREPACK_ENABLE_DOWNLOAD_PROMPT=0 +RUN corepack enable + +# Gems live in a named volume so `bundle install` survives a container restart +# and does not fight the bind-mounted source tree. +# +# RUBYOPT is not cosmetic. Ruby 3.1 auto-activates its own bundled +# error_highlight 0.3.0 before Bundler runs, and the Gemfile's development group +# asks for >= 0.4.0, which resolves to 0.5.1 -- so `bin/rails` dies at +# config/boot.rb with "You have already activated error_highlight 0.3.0". +# Disabling the startup require lets Bundler activate the locked version. +# +# Neither the production image nor CI hits this, because both exclude the +# development group (BUNDLE_WITHOUT); this container is the only place that +# installs it, and it has to, because rubocop and i18n-tasks live there. +# +# Upgrading RubyGems does NOT fix it -- `gem update --system 3.4.22` was tried +# and the conflict is unchanged. The flag is the whole fix. +ENV BUNDLE_PATH=/bundle \ + BUNDLE_BIN=/bundle/bin \ + PATH=/bundle/bin:$PATH \ + RUBYOPT=--disable-error_highlight + +WORKDIR /app diff --git a/docker/db-init.sql b/docker/db-init.sql new file mode 100644 index 0000000..5229a53 --- /dev/null +++ b/docker/db-init.sql @@ -0,0 +1,12 @@ +-- The compose environment creates openipc_development and grants www on it. +-- The suite needs more than that: openipc_test, plus one database per Minitest +-- worker (openipc_test-0, openipc_test-1, ... -- the suite is parallelized +-- across cores), and those are created by Rails at run time under the www user. +-- +-- The backslash escapes the underscore so it is matched literally rather than +-- as MySQL's single-character wildcard; without it the grant would also cover +-- databases like "openipcXfoo". +CREATE DATABASE IF NOT EXISTS openipc_test + CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; +GRANT ALL PRIVILEGES ON `openipc\_%`.* TO 'www'@'%'; +FLUSH PRIVILEGES; From c317351a31ee7b9af39cb0f083ed259e86705418 Mon Sep 17 00:00:00 2001 From: AI Dev Date: Tue, 25 Aug 2026 15:07:43 +0000 Subject: [PATCH 2/2] Have the dev container write files you can delete The stack as first written ran as root, so everything it put into the bind-mounted tree -- app/assets/builds, log/, tmp/, the output of yarn build:fonts -- landed on the host owned by root. Using it for one afternoon left public/fonts/ owned by nobody:nogroup and a git checkout that could not remove it: "warning: unable to unlink", twenty times. The image now creates a user and runs as it, and chowns /bundle to it so bundler can still install into the volume. The ids come from DEV_UID/DEV_GID build args defaulting to 1000, which is the usual first user on a Linux desktop; anyone else writes them into a .env beside compose.yaml, which Compose reads on its own and which is now gitignored. The node_modules named volume goes. A named volume is created root-owned and Docker seeds it from the image, so an empty one over a path the image does not have stays root-owned and yarn fails on the first link step with EACCES. /bundle does not have this problem precisely because the image creates and chowns it before the volume is populated from it. node_modules lives in the bind mount instead, where it belongs to the developer and is already gitignored. Migrating an existing checkout needs one command, since the old root-owned files resist the new user: docker compose run --rm --user 0:0 web chown -R "$(id -u):$(id -g)" \ /app/.yarn /app/node_modules /app/app/assets/builds /app/log /app/tmp Verified: container reports uid 1002, a file it touches in the tree reads ai:ai on the host and deletes without sudo, and the suite is unchanged at 231 runs, 873 assertions, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015hvzXBErEjEGMyRme2K3hi --- .gitignore | 3 +++ compose.yaml | 26 ++++++++++++++++++++++---- docker/Dockerfile.dev | 20 ++++++++++++++++++++ 3 files changed, 45 insertions(+), 4 deletions(-) diff --git a/.gitignore b/.gitignore index c7e5580..1678e30 100644 --- a/.gitignore +++ b/.gitignore @@ -57,3 +57,6 @@ # tools/webui-gallery installs its own node_modules; the tool is not part of the app build. /tools/webui-gallery/node_modules + +# Local overrides for the dev compose stack (DEV_UID/DEV_GID); never committed. +/.env diff --git a/compose.yaml b/compose.yaml index 73d365e..a0cc970 100644 --- a/compose.yaml +++ b/compose.yaml @@ -47,6 +47,18 @@ services: # tree, and a root context would ship node_modules to the daemon. context: ./docker dockerfile: Dockerfile.dev + args: + DEV_UID: ${DEV_UID:-1000} + DEV_GID: ${DEV_GID:-1000} + # Matched to the build args above so files written into the bind-mounted + # tree belong to you. If `id -u` is not 1000, put your ids in a .env file + # beside this one and rebuild: + # + # printf 'DEV_UID=%s\\nDEV_GID=%s\\n' "$(id -u)" "$(id -g)" > .env + # docker compose build web + # + # Compose reads .env automatically; it is gitignored. + user: ${DEV_UID:-1000}:${DEV_GID:-1000} command: bash -c 'bundle check || bundle install; exec bin/rails server -b 0.0.0.0 -p 3010' environment: # Setting this is what makes config/database.yml connect over TCP instead @@ -57,10 +69,17 @@ services: - '3010:3010' volumes: - .:/app + # /bundle is a named volume rather than part of the tree, so `bundle + # install` survives a rebuild. It works with the non-root user above + # because the image creates and chowns /bundle before the volume is + # populated from it -- Docker seeds an empty named volume from the image, + # ownership included. + # + # node_modules deliberately gets no such volume: an empty named volume is + # created root-owned with nothing in the image to seed it from, so yarn + # would fail with EACCES on the first link step. It lives in the bind mount + # instead, where it belongs to you and is already gitignored. - bundle:/bundle - # node_modules is Linux-native and must not be shadowed by, or written - # back to, whatever the host happens to have installed. - - node-modules:/app/node_modules depends_on: db: condition: service_healthy @@ -68,4 +87,3 @@ services: volumes: bundle: db-data: - node-modules: diff --git a/docker/Dockerfile.dev b/docker/Dockerfile.dev index 9a88b4b..77f9996 100644 --- a/docker/Dockerfile.dev +++ b/docker/Dockerfile.dev @@ -71,4 +71,24 @@ ENV BUNDLE_PATH=/bundle \ PATH=/bundle/bin:$PATH \ RUBYOPT=--disable-error_highlight +# Run as the developer, not as root. +# +# Everything this container writes into the bind-mounted tree -- app/assets/builds, +# log/, tmp/, and anything `yarn build:fonts` produces -- lands on the host with +# the writer's ownership. As root that means files the developer cannot delete or +# edit, which is how public/fonts/ ended up owned by nobody:nogroup the first time +# this stack was used in anger. +# +# The uid has to exist in the image and own /bundle, or bundler cannot install +# into the volume. 1000 is the usual first user on a Linux desktop; override with +# DEV_UID/DEV_GID when it is not (see compose.yaml). +ARG DEV_UID=1000 +ARG DEV_GID=1000 +RUN groupadd --gid ${DEV_GID} dev 2>/dev/null || true \ + && useradd --uid ${DEV_UID} --gid ${DEV_GID} --create-home --shell /bin/bash dev 2>/dev/null || true \ + && mkdir -p /bundle \ + && chown -R ${DEV_UID}:${DEV_GID} /bundle + +USER ${DEV_UID}:${DEV_GID} + WORKDIR /app