Huge source map since 4.0.22

[stevenwdv]

Since v4.0.22 the source map (-gsource-map) for our application has become huge.
In v4.0.21, ourApp.wasm.map was 8.3 MiB, but in v4.0.22 and v4.0.23 it's 1.9 GiB, which hangs (Firefox) / crashes (Chrome) the browser when opening the console. (For chrome, we can use DWARF, but for Firefox this is not an option.) I presume that the cause for this is #25870.

Is there a possibility that this could be reduced, or maybe to add a setting to exclude certain large sections or truncate long names? Or maybe use mangled names (see below)?

Some observations:

• When compressed with a simple gzip ourApp.wasm.map, ourApp.wasm.map.gz is just 24 MiB, meaning there's a lot of duplicate text in there.
• I thinks the cause is probably the same as for [WebAssembly] Huge symbol name section in debug build llvm/llvm-project#126919, meaning demangling C++ templated identifiers lead to a size explosion. Indeed, a quick cat ourApp.wasm.map spams my screen with such identifiers. We're already using -Wl,--no-demangle to mitigate the LLVM issue, but this doesn't seem to affect the source map.
• When counting word frequency, I see that our namespace occurs 33M times and std occurs 29M times. The likely culprit, rxcpp, occurs 10M times (we want to ditch it but don't have time).

[sbc100]

I don't think #25298 is the cause, as it only effect table and memory (of which there are exactly one of each).

More likely is #25870 which added new info the source map. @aheejin @dschuff know more about this.

[stevenwdv]

@sbc100 Ah I see, it seems the wrong PR was linked in ChangeLog.md

[dschuff]

Ah, thanks for the heads-up on the changelog, we can fix that.

#25870 definitely made source maps bigger. The hope is that many users could switch from having source maps plus a name section in the binary (https://emscripten.org/docs/porting/Debugging.html#overview-emitting-and-controlling-debug-information) to just source maps, which would effectively move that information from the binary to the source map without making things any bigger overall. But if this is causing issues with browser devtools then maybe we need to be more flexible there. Does your binary have a name section? Does your use case work when using a name section instead of names in the source map (or did you just not have names at all before?)

I think the reason -Wl,--no-demangle doesn't affect source maps is that it's a linker flag (which affects the generation of the name section) but the source map generator gets its information from DWARF, which IIRC uses the original names. I imagine it would be possible to use mangled names in the source map, if we were really sure that it would help (my real preference would be for the browser devtools to be more scalable :))

[dschuff]

BTW, you can file an issue at issues.chromium.org about the Chrome devtools crashing with large source maps. If you post it here I can try to make sure it gets routed appropriately.

[aheejin]

The support of names field (#25870 and its reland #25928) definitely increased the source map size, but the 220x increase in your case sounds extreme. My measurement of a mid-sized C++ program (Binaryen) was more like 3x range. Different programs have different characteristics and Binaryen may not be a representative case.

1. Can you possibly share what your binary (with the name section) or source map?
2. Or if it's not possible, can you share the result of wasm-objdump -h or llvm-objdump -h that shows each section size?
3. Just checking, does the source map include the info on the same sets of source files as before?

If the increase in the source map size is generally prohibitively large, I think we can or should put it behind an option. But I'd like to check if it can really cause 220x increase first.

[stevenwdv]

@dschuff

The hope is that many users could switch from having source maps plus a name section in the binary

I see. Out of curiosity, what's the reason for this? A more standard approach? Better browser support? (Do current browsers support the name section?)

Does your binary have a name section?

Yes, it is 54 MiB. Without --no-demangle it's 2.1 GiB and the toolchain starts to malfunction. See again llvm/llvm-project#126919.

Does your use case work when using a name section instead of names in the source map (or did you just not have names at all before?)

You mean if Firefox shows function names with v4.0.21? Hm maybe not, I just see stuff like (wasmcall) file.cpp:123 in the stack in the debugger. I'm not sure if it worked before because sometimes results change. So, if this change can fix that, that's great! Btw, for file names to even work it seems like I need -g in addition to -gsource-map, I guess because "the source map generator gets its information from DWARF"?

I imagine it would be possible to use mangled names in the source map, if we were really sure that it would help (my real preference would be for the browser devtools to be more scalable :))

I would strongly prefer the former! (At least behind a flag.) (And maybe in addition to the latter, if possible.) A gigabite file is just not workable. Templated types can become really long and repetitive: often a long templated type gets repeated, e.g. frobber<foo_very_long_name, frob_adapter<foo_very_long_name>>, which leads to an explosion in our case. Mangling eliminates these repetitions. I will say that there seem to be some functions in our codebase that are especially bad, and I may look into mitigating this, but that'll probably not fully fix the issue.

BTW, you can file an issue at issues.chromium.org about the Chrome devtools crashing with large source maps. If you post it here I can try to make sure it gets routed appropriately.

I can still do that as well, I'll file it later. https://issues.chromium.org/issues/476025609

---

@aheejin

The support of names field [...] definitely increased the source map size, but the 220x increase in your case sounds extreme.

It is, but not unexpected given llvm/llvm-project#126919. See what I mention in reply to "I imagine it would be possible to use mangled names in the source map [...]" above.

Can you possibly share what your binary (with the name section) or source map?

I can. I can't attach it, but I uploaded it here. This is a Debug build with -g -gseparate-dwarf -gsource-map -Wl,--no-demangle compiled with v4.0.23.

Or if it's not possible, can you share the result of wasm-objdump -h or llvm-objdump -h that shows each section size?

For completeness:

Details

$ "$EMSDK/upstream/bin/llvm-objdump" --section-headers pepWeblib.wasm

pepWeblib.wasm: file format wasm

Sections:
Idx Name                Size     VMA      Type
  0 TYPE                00000645 0000000b 
  1 IMPORT              00000c53 00000653 
  2 FUNCTION            00028972 000012aa 
  3 TABLE               00000009 00029c1e 
  4 TAG                 00000003 00029c29 
  5 GLOBAL              00000d9d 00029c2f 
  6 EXPORT              00000457 0002a9cf 
  7 START               00000001 0002ae28 
  8 ELEM                000102d5 0002ae2d 
  9 DATACOUNT           00000001 0003b104 
 10 CODE                01646e00 0003b10a TEXT
 11 DATA                00248b84 01681f0f DATA
 12 name                034a5ba9 018caa98 
 13 sourceMappingURL    00000013 04d70648 
 14 target_features     000000a1 04d7066f 
 15 external_debug_info 0000001a 04d70722

See also the answer to "Does your binary have a name section?" above.

Just checking, does the source map include the info on the same sets of source files as before?

What do you mean "as before"? Like with llvm/llvm-project#126919 ? There has been a lot of development but roughly yes.

Or do you mean like with the v4.0.21 build? In that case yes: I switched back-and-forth between EMSDK versions to test.

If the increase in the source map size is generally prohibitively large, I think we can or should put it behind an option.

If it's mostly specific to highly-templated code. You could also consider something like -Wl,--no-demangle for the source map.

[stevenwdv]

@dschuff I filed it: https://issues.chromium.org/issues/476025609

[stevenwdv]

As I see I noted in llvm/llvm-project#126919, the large symbol names contain a lot of logic that I'd expect to be evaluated at compile time, e.g. std::__2::conditional, std::is_same_v, and std::__2::invoke_result. What's up with that?

[stevenwdv]

I will say that there seem to be some functions in our codebase that are especially bad, and I may look into mitigating this, [...]

FYI, by using two .as_dynamic()s (which convert rxcpp::observable<T, SourceOperator> to rxcpp::observable<T>) in a single rxcpp::observable chain I was at least able to reduce 1.9 GiB it to 742 MiB.