Comet throws `java.lang.NegativeArraySizeException`

[sandugood]

Describe the bug

While testing out Comet on a rather large pipeline, querying Iceberg tables, the execution dropped with an error:

Py4JJavaError: An error occurred while calling o12640.createOrReplace.
: java.lang.NegativeArraySizeException: -954238687
	at org.apache.comet.shaded.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:46)
	at org.apache.comet.serde.operator.CometIcebergNativeScan$.serializePartitions(CometIcebergNativeScan.scala:1025)
	at org.apache.spark.sql.comet.CometIcebergNativeScanExec.serializedPartitionData$lzycompute(CometIcebergNativeScanExec.scala:111)
	at org.apache.spark.sql.comet.CometIcebergNativeScanExec.org$apache$spark$sql$comet$CometIcebergNativeScanExec$$serializedPartitionData(CometIcebergNativeScanExec.scala:86)
	at org.apache.spark.sql.comet.CometIcebergNativeScanExec.commonData(CometIcebergNativeScanExec.scala:114)

Note that during Spark session building we distribute Python environment to the executors by:

"spark.archives": "/path/to/venv3.10.12.tar.gz#venv"

And build the env like this (interactively inside JupyterNotebook cell):

! conda create -y -n myenv python=3.10.12
! conda run -n myenv pip install -r ../requirements.txt
! conda run -n myenv pip install conda-pack
! conda run -n myenv conda-pack -o venv3.10.12.tar.gz

Also, when running with log level set to "WARN", getting such messages:
WARN DAGScheduler: Broadcasting large task binary with size 79.0 MiB

It seems that it might be related to how Comet distributes Python environment, or bags whole plan into a byteArray (which might exceed threshold)

Additional info:

• Iceberg V2
• Positional (row-level) deletes.
• MOR tables involved in the pipeline

Steps to reproduce

No response

Expected behavior

Comet doesn't crash on this query.

Additional context

No response