osm2pgsql to pmtiles / Looking for a simple way to generate community map and data projects

[tordans]

Maybe someone already has this set up and can share something.
And if not, maybe this is something to think about…

Kontext:
It is very easy to host static websites (GitHub Pages etc.) and static files (S3 etc.). It is a lot harder to host a tile server (etc.).
We have more and more OSM Community projects that create topic-specific data for vector maps and exports.
Right now, the tooling that osm2pgsql provides for this is still quite complex (for good reasons and optimized for those reasons).
Constraint: To keep things simple, I am only looking for data updates maybe every 6-12 hours or daily, but not minutely.

Question:
What is the simplest setup I can run to create daily PMtiles for a data set using osm2pgsql to handle my data processing?

Examples:

• TILDA https://github.com/FixMyBerlin/tilda-geo
  Daily PBF > Osmium tag filter > osm2pgsql > DB > DB-Functions (for Generalization) > Martin > Vectortile Endpoints > Filesystem Cache
  This system works great for TILDA but is way too complex for a hobby project.

• OpenRailwayMap Vector https://github.com/hiddewie/OpenRailwayMap-vector
  I did not investigate the system in detail but it looks like they use Martin to create mbtiles and then create pmtiles from them.

• Streuobstwiesen Karte https://github.com/ohrie/portal-streuobst, https://portal-streuobst.de/karte/#6/51.166/10.452
  AFAIK the process is PBF > Osmium > Gdal > Tippecanoe

• Americana https://wiki.openstreetmap.org/wiki/OpenStreetMap_Americana
  Uses Planetiler.

Ideal setup ideas:

• Ideally, I would like to use PMtiles, which I can just put somewhere and reference from my vector map.
• Ideally, I would like to use tippecanoe to create the PMtiles because for most projects, the Tippecanoe magic of generalization is enough to get a usable result.
• Ideally, there would be a way to pass the data from OSM2PGSQL directly to Tippecanoe or with a minimal and storage-efficient intermediate step.
• It would be perfect if there were a way to run the OSM2PGSQL process without relying on a database at all; or one that I can create self-contained in the processing Docker image and that can be deleted after the PMtile files are created.

Output formats

Some of those processes above only focus on the vector tile output (using different methods). That is a very valid constraint to simplify the process when the result is "just" a visual map. But many of the projects I see coming up can and should also consider the exportable data form their pipeline part of their assets. This is the great thing we have when running the current osm2pgsql setup, we already have the database to export from. This is an aspect we could ideally preserve. The question is, how to do this in a file based mindset. Pmtiles are not the right format here, but GeoParquet files look like a good candidate to fill this gap.

---

I know quite a few pieces of this puzzle are not how things are now. But we will be seeing many new vector-based map projects in the near future, and many of them could run a similar pipeline. It would be great if more OSM projects would use a similar approach so contributing will be easy, and the projects work under similar constraints but also have similar capabilities.

[pnorman]

The goal of osm2pgsql is to import OSM data into a postgis database and keep it updated. If your ideal setup doesn't have a database and you're only after daily updates, why are you starting with osm2pgsql?

You certainly can create an osm2pgsql database and export a planet's worth of vector tiles, but why? It's going to take much longer than a setup optimized for generating the planet like tilemaker or planetiler.

I happen to believe the flex API is more powerful than the configuration options other software has, and the combination of the power of the flex API and PostGIS gives an osm2pgsql-based toolchain far more power. This does not matter for someone looking for a simple way to generate tiles.

[tordans]

The goal of osm2pgsql is to import OSM data into a postgis database and keep it updated.

Part of this discussion is to challenge and evolve this framing. I am quite sure that the community is looking for a solution for the cases described; and we can either evolve existing tools or build new once… – Both is fine; lets talk about both.

A bit broder inspiration on this: https://en.wikipedia.org/wiki/Marketing_myopia

If your ideal setup doesn't have a database and you're only after daily updates, why are you starting with osm2pgsql?

As you said, the tooling osm2pgsql provides is great. Personally I spend a lot of time learning it. It would be great to be able to start a project with the same tooling and return pmtiles and be able to transition to rending from DB (eg. when more .sql is needed) and then to minutely updates at some point, when needed. It would allow projects to transition more easily from an experiment to a production ready app.

You certainly can create an osm2pgsql database and export a planet's worth of vector tiles…

It would be great if we shared more about the process for this: How?…
I listed a few examples and all used different approaches to this problem.

…, but why?

I tried to explain that above. Pmtiles opens up hosting vector data to a whole new user group with much lower cost and especially much simpler – just one file vs. managing servers. My understanding is that GeoParquet does the same but for the "data for analysis" branch of the story.

It's going to take much longer than a setup optimized for generating the planet like tilemaker or planetiler.

One key difference to me is: For now, those apps are focussed on generating data optimized for the usage on a map style. However, IMO the big advantage of osm2pgsql today is that it does this but also provides a way to export opinionated OSM data. Which is something that I hope many of those coming topic based OSM data processing pipelines will provide. Not just show data but also allow exporting/using them as cleaned and processed OSM data. https://en.osm.town/@tordans/112637637684642696 has a few quotes on this topic which I find myself getting back to again and again.

For example: https://radinfra.de/ is great to show processed bike infrastructure data from OSM. But the export (which in our case is not public) is just as or more valuable. Same for https://www.osm-verkehrswende.org/cqi/map/?anzeige=cqi&map=14.9%2F52.4989%2F13.4166&filters=usable-yes where the visual map is just one thing, but the exported, processed data would be just as useful for analysis and routing use cases.

[lonvia]

No, just no. That's feature creep at its worst. osm2pgsql is for processing data with Postgresql. That's so tightly coupled in its design that we are not even considering support for other SQL databases. And that would be much closer to what you are suggesting.

Your reference to "Marketing myopia" is naive at best. You are not suggesting something that "opens new markets". Your are explicitly marketing for a user group that is neither willing nor able to make the necessary investments. You are pretty much asking us to make your life easier for nothing in return.

If you are willing to back your feature requests with the necessary investments (we are talking about a high six-digit figure here), then by all means get in touch and we can talk.

Otherwise I refer to the roadmap which explains well the current scope of osm2pgsql and close this discussion.