Project: S3779 LZW ColGroup | ASML by MasterBrain2000 · Pull Request #2560 · apache/systemds

MasterBrain2000 · 2026-07-19T19:28:04Z

Group Project for the AMLS Module

Closes apache#2420.

…nd downstream CLA operations

janniklinde · 2026-07-20T08:35:54Z

Your code does not build (see error logs). Could you please fix those issues to have a prototype that compiles?

janniklinde · 2026-07-29T07:44:29Z

@MasterBrain2000 @m-ollka @Mancer1 could you please address the issues causing the tests to fail? Thanks

Mancer1 · 2026-08-03T07:41:02Z

@janniklinde, all the build problems have been addressed.

codecov · 2026-08-03T09:56:03Z

Codecov Report

❌ Patch coverage is 89.80100% with 82 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.51%. Comparing base (d5f6950) to head (5d0b8b4).
⚠️ Report is 27 commits behind head on main.

Files with missing lines	Patch %	Lines
...ss/colgroup/ColGroupPiecewiseLinearCompressed.java	89.32%	34 Missing and 39 partials ⚠️
...ress/colgroup/functional/PiecewiseLinearUtils.java	92.22%	3 Missing and 4 partials ⚠️
...he/sysds/runtime/compress/colgroup/ColGroupIO.java	60.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2560      +/-   ##
============================================
- Coverage     71.61%   71.51%   -0.11%     
- Complexity    50132    50658     +526     
============================================
  Files          1614     1631      +17     
  Lines        193986   196282    +2296     
  Branches      37935    38267     +332     
============================================
+ Hits         138925   140373    +1448     
- Misses        44155    44910     +755     
- Partials      10906    10999      +93

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Mancer1 · 2026-08-03T12:01:29Z

@janniklinde, for code coverage, aren't tests considered part of it? We have some helper functions from the old DP algorithm, and we used to test them against the new algorithm. These DP helper functions are not used anywhere other than the tests we wrote.

Mancer1 · 2026-08-03T12:15:32Z

@janniklinde, How much code coverage is accepted? We have some null functions that are required to be implemented but never called.

janniklinde · 2026-08-03T13:17:53Z

Generally, all functions that are expected to provide a working implementation should be tested, @Mancer1. Coverage also includes nested function calls, so you should be able to achieve high coverage by testing the publicly available functions of the compressed column group. Please use the tests for existing AColGroup implementations as a reference.

Also, please avoid reimplementing helper functionality that is already available in the test utilities, such as random matrix generation.

…o improve coverage

# Conflicts: # src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupPiecewiseLinearCompressed.java # src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupPiecewiseLinearCompressedOperationsTest.java

Mancer1 · 2026-08-07T19:07:24Z

@janniklinde, we have improved the coverage to 95%. I hope that suffices

Mancer1 · 2026-08-08T11:37:46Z

@janniklinde, we have already reformatted our files that we edited. The files that java Codestyle check is showing to reformat are those not part of our PR.

rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$  dev/format-changed.sh
All PR-edited Java lines are correctly formatted.
rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$ git status
On branch main
Your branch is up to date with 'origin/main'.

So, should we leave it be, since we were working on a stale version of main?

janniklinde · 2026-08-10T07:57:23Z

You can simply run the following @Mancer1:

git fetch upstream
dev/format-changed.sh upstream/main

Mancer1 · 2026-08-10T08:09:09Z

You can simply run the following @Mancer1:
git fetch upstream
dev/format-changed.sh upstream/main

@janniklinde, I believe I tried it before. It didn't affect anything.

rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$ git fetch upstream
dev/format-changed.sh upstream/main
remote: Enumerating objects: 299, done.
remote: Counting objects: 100% (172/172), done.
remote: Compressing objects: 100% (90/90), done.
remote: Total 299 (delta 105), reused 82 (delta 82), pack-reused 127 (from 2)
Receiving objects: 100% (299/299), 262.07 KiB | 507.00 KiB/s, done.
Resolving deltas: 100% (122/122), completed with 16 local objects.
From https://github.com/apache/systemds
 * [new branch]            dependabot/github_actions/docker/login-action-4.6.0 -> upstream/dependabot/github_actions/docker/login-action-4.6.0
   7a0e5ca0ac..9dccbc3e60  main       -> upstream/main
All PR-edited Java lines are correctly formatted.
rice@Arnor:~/MasterBrain2000-Project-S3779-LZW-ColGroup$ git status
On branch main
Your branch is up to date with 'origin/main'.
```

janniklinde

Thanks for the updates. I left comments in the code which should still be fixed. Most of them are minor and easy to patch.

Please address those remaining issues to pass the project.

janniklinde · 2026-08-10T12:52:44Z

+	/**
+	 * Utility methods for piecewise linear compression of matric columns supports compression used the segmented least
+	 * squares algorithm which is implemented with dynamic programming and a successive method, which puts all values in
+	 * a segment till the target loss is exceeded
+	 */


Comment not up to date, either remove or update and put above class definition

janniklinde · 2026-08-10T13:00:52Z

+	public static SegmentedRegression compressSuccessivePiecewiseLinear(double[] column, CompressionSettings cs) {
+		// compute Breakpoints for a Column with a sukzessive breakpoints algorithm
+
+		final List<Integer> breakpointsList = computeBreakpointSuccessive(column, cs);
+		final int[] breakpoints = breakpointsList.stream().mapToInt(Integer::intValue).toArray();
+
+		// get values for Regression
+		final int numSeg = breakpoints.length - 1;
+		final double[] slopes = new double[numSeg];
+		final double[] intercepts = new double[numSeg];
+
+		// Regress per Segment
+		for(int seg = 0; seg < numSeg; seg++) {
+			final int segstart = breakpoints[seg];
+			final int segEnd = breakpoints[seg + 1];
+			final double[] line = regressSegment(column, segstart, segEnd);
+			slopes[seg] = line[0];
+			intercepts[seg] = line[1];
+		}
+		return new SegmentedRegression(breakpoints, slopes, intercepts);
+	}


These comments are unnecessary. I'd prefer a clear documentation comment for the function itself.

janniklinde · 2026-08-10T13:03:39Z

+	/**
+	 * computes the segment cost
+	 *
+	 * @param column column values
+	 * @param start  start index
+	 * @param end    end index
+	 * @return SSE of the regression line over the segment
+	 */
+	public static double computeSegmentCost(double[] column, int start, int end) {
+		final int segSize = end - start;
+		if(segSize <= 1)
+			return 0.0;
+
+		final double[] ab = regressSegment(column, start, end);
+		final double slope = ab[0];
+		final double intercept = ab[1];
+
+		double sse = 0.0;
+		for(int i = start; i < end; i++) {
+			double err = column[i] - (slope * i + intercept);
+			sse += err * err;
+		}
+		return sse;
+	}


Can be removed (including the test)

janniklinde · 2026-08-10T13:19:05Z

+	 * @param cs         compression settings to define the target loss, which should be considered
+	 * @return a piecewise linear compressed column group
+	 */
+


Remove line

janniklinde · 2026-08-10T13:20:22Z

+	public static AColGroup compressPiecewiseLinearFunctional(IColIndex colIndexes, MatrixBlock in,
+		CompressionSettings cs) {
+
+		final int numRows = in.getNumRows();
+		final int numCols = colIndexes.size();
+		int[][] breakpointsPerCol = new int[numCols][];
+		double[][] slopesPerCol = new double[numCols][];
+		double[][] interceptsPerCol = new double[numCols][];
+
+		for(int col = 0; col < numCols; col++) {
+			final int colIdx = colIndexes.get(col);
+			double[] column = PiecewiseLinearUtils.getColumn(in, colIdx);
+			PiecewiseLinearUtils.SegmentedRegression fit = PiecewiseLinearUtils
+				.compressSuccessivePiecewiseLinear(column, cs);
+			breakpointsPerCol[col] = fit.getBreakpoints();
+			interceptsPerCol[col] = fit.getIntercepts();
+			slopesPerCol[col] = fit.getSlopes();
+
+		}
+		return ColGroupPiecewiseLinearCompressed.create(colIndexes, breakpointsPerCol, slopesPerCol, interceptsPerCol,
+			numRows);
+
+	}


Incorrect if cs.transposed = true, ther e you need to iterate over rows

janniklinde · 2026-08-10T14:06:27Z

+	public AColGroup sort() {
+		return this;
+	}


Throw, only correct for already sorted data

janniklinde · 2026-08-10T14:06:46Z

+	public AColGroup reduceCols() {
+		return null;
+	}


janniklinde · 2026-08-10T14:10:04Z

Remove legacy text

janniklinde · 2026-08-10T14:12:38Z

Remove DP tests

janniklinde · 2026-08-10T14:18:03Z

A few recurring patterns I noticed here:

Several tests only verify return types, non-null values, or that unsupported methods throw

Some expected results derived from the implementation itself instead of being computed independently

Error tolerances sometimes very loose relative to input range

Missing edge cases (e.g., non-commutative ops, transposed input, zero target loss, serialization through the normal ColGroupIO)

Leftover DP tests

MaryAbdull and others added 11 commits July 19, 2026 15:51

[SYSTEMDS-3543] Piece-wise linear compression of column groups

d767502

Closes apache#2420.

PiecewiseLinearUtils.computeBreakpointSuccessive is compressed in O(n)

d73a371

Old DP algorithm and its usages are removed

f442491

add DDCLZW enum value to ColGroupType

6dea132

new ColGroupPiecewiseLinearCompressed methods; fix testComputeSUm

5bd5180

new replace & unitaryOperation

0263992

ColGroupIO support

f3fe67f

[Minor] fixed mistake in previous commit

ff38235

Performance tests and Benchmarks for sucussive Piece wise algorithm a…

5e50b92

…nd downstream CLA operations

added operation tests

37ef00d

using tab instead of spaces for intentations

86f671a

github-project-automation Bot added this to SystemDS PR Queue Jul 19, 2026

github-project-automation Bot moved this to In Progress in SystemDS PR Queue Jul 19, 2026

Mancer1 added 12 commits August 2, 2026 21:29

Implement missing abstract method in ColGroupPiecewiseLinearCompressed

07be457

ci: configure GPG loopback pinentry for release script workflow

5cc6487

ci: configure GPG loopback pinentry for release script workflow

10e86e8

ci: configure GPG loopback pinentry for release script workflow

f99b824

ci: configure GPG loopback pinentry for release script workflow

4a9a13a

Implement missing abstract method in ColGroupPiecewiseLinearCompressed

29153b1

added missing import

de890dc

added all missing abstract methods

4da1965

ci: restore release-scripts.yml workflow to original state

e51a38f

added the missing licenses and imports

5cc8a39

style: fix checkstyle violations in test classes

82c8bd9

added the missing ValueFunction import

1298867

style: apply eclipse line formatting fixes via format-changed.sh

02b90bf

Mancer1 and others added 7 commits August 3, 2026 16:43

Remove all DP functions and using testUtils helper function instead t…

f3d419d

…o improve coverage

deleting unused imports, and formatting

bfdbcf0

added skeleton for remaining 0% coverage functions

1bf3737

updated skeleton for 0% coverage functions

69a1013

testing write

02cf453

# Conflicts: # src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupPiecewiseLinearCompressed.java # src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupPiecewiseLinearCompressedOperationsTest.java

tests 16-30

7004f24

tests 01-15

83d9bee

Mancer1 force-pushed the main branch from abfa731 to 83d9bee Compare August 6, 2026 22:29

Mancer1 added 2 commits August 7, 2026 00:30

Add unit tests and coverage fixes and some comments

448642c

correcting Java codestyle

5d0b8b4

janniklinde self-requested a review August 10, 2026 11:10

janniklinde requested changes Aug 10, 2026

View reviewed changes

github-project-automation Bot moved this from In Progress to In Review in SystemDS PR Queue Aug 10, 2026

Uh oh!

Conversation

MasterBrain2000 commented Jul 19, 2026

Uh oh!

janniklinde commented Jul 20, 2026

Uh oh!

janniklinde commented Jul 29, 2026

Uh oh!

Mancer1 commented Aug 3, 2026

Uh oh!

codecov Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Mancer1 commented Aug 3, 2026

Uh oh!

Mancer1 commented Aug 3, 2026

Uh oh!

janniklinde commented Aug 3, 2026

Uh oh!

Mancer1 commented Aug 7, 2026

Uh oh!

Mancer1 commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janniklinde commented Aug 10, 2026

Uh oh!

Mancer1 commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janniklinde left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Aug 3, 2026 •

edited

Loading

Mancer1 commented Aug 8, 2026 •

edited

Loading

Mancer1 commented Aug 10, 2026 •

edited

Loading