feat(server): Remote Materialization by aniketpalu · Pull Request #6649 · feast-dev/feast

aniketpalu · 2026-07-28T11:11:07Z

Summary

Supersedes #6590 (clean single-commit history with proper DCO).
Add ?async=true query parameter support to the existing /materialize and /materialize-incremental endpoints. When set, materialization runs in a background thread and the endpoint returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Also adds ?force=true to allow operators to override stuck MATERIALIZING state (e.g. after a crashed SparkApplication pod).

Client-side (SDK)

store.materialize(..., remote=True) delegates to the feature server
(URL/TLS derived from online_store config)
wait=False returns immediately after triggering
store.poll_materialization() for status polling via registry state

Server-side

?async=true on existing /materialize and /materialize-incremental
?force=true to skip the 409 concurrency guard
Four module-level helpers extracted for testability

Registry

SQL and Snowflake apply_materialization() now set FV state to
AVAILABLE_ONLINE (parity with file-based registry)

Which issue(s) this PR fixes

Addresses #4526

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Misc

When ?async=true is passed to /materialize and /materialize-incremental, the endpoint fires off materialization in a background thread and returns 202 Accepted immediately. Concurrent requests for an already-MATERIALIZING FV are rejected with 409. Without ?async=true, behavior is unchanged. Client-side additions: - remote=True param on store.materialize() / materialize_incremental() to delegate to the feature server (URL/TLS from online_store config) - wait=False support with store.poll_materialization() for status polling - FeatureView state set to MATERIALIZING before 202, reset on failure Server-side additions: - ?async=true on existing /materialize and /materialize-incremental - ?force=true to override stuck MATERIALIZING state - Four module-level helpers for testability: _authorize_materialize_views, _check_already_materializing, _update_fv_state, _parse_materialize_timestamps Registry fixes: - SQL and Snowflake registries now set FV state to AVAILABLE_ONLINE in apply_materialization() (parity with file-based registry) Addresses feast-dev#4526 Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

codecov-commenter · 2026-07-28T11:23:39Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 9.60452% with 160 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.66%. Comparing base (a1e6fc2) to head (059c2d2).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
sdk/python/feast/feature_server.py	11.22%	87 Missing ⚠️
sdk/python/feast/feature_store.py	8.21%	61 Missing and 6 partials ⚠️
sdk/python/feast/infra/registry/snowflake.py	0.00%	3 Missing ⚠️
sdk/python/feast/infra/registry/sql.py	0.00%	3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6649      +/-   ##
==========================================
- Coverage   46.77%   46.66%   -0.11%     
==========================================
  Files         414      414              
  Lines       50191    50339     +148     
  Branches     7181     7208      +27     
==========================================
+ Hits        23475    23489      +14     
- Misses      25077    25207     +130     
- Partials     1639     1643       +4

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`47.97% <9.60%> (-0.13%)`	⬇️

Files with missing lines	Coverage Δ
sdk/python/feast/infra/registry/snowflake.py	`38.92% <0.00%> (-0.21%)`	⬇️
sdk/python/feast/infra/registry/sql.py	`57.36% <0.00%> (-0.25%)`	⬇️
sdk/python/feast/feature_store.py	`42.96% <8.21%> (-1.44%)`	⬇️
sdk/python/feast/feature_server.py	`56.45% <11.22%> (-5.40%)`	⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4efb86c...059c2d2. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…ore is remote When online_store.type == remote, materialize() and materialize_incremental() POST to the feature server with ?async=true (fire-and-forget) instead of running the engine locally. Optional force=True maps to ?force=true for stuck MATERIALIZING recovery. URL/TLS/auth come from online_store config. Shared _delegate_remote_materialize() builds query params and posts; RemoteComputeEngine remains a follow-up for provider-layer remoting. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

Remove server-side MATERIALIZING pre-set before 202 which conflicted with store.materialize() state machine (MATERIALIZING → MATERIALIZING rejected). Async now accepts and runs materialize in the background; store owns transitions. ?force=true resets stuck MATERIALIZING FVs to GENERATED so normal materialize can proceed. Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

ntkathole · 2026-07-29T11:24:36Z

+        force: bool = False,
+    ) -> None:
+        """Fire-and-forget POST to feature server with ?async=true."""
+        query_params = {"async": "true"}


may be allow user to pass param, There's no way for an SDK user to do synchronous materialization now.

Agreed — added run_async: bool = True on materialize() / materialize_incremental().

Default run_async=True: remote topology POSTs with ?async=true, returns after 202

run_async=False: POSTs without async and blocks until the server finishes sync materialization

force=True still requires run_async=True (force only applies on the async path)

franciscojavierarceo

I found two blocking correctness issues:

The async concurrency guard is check-then-act. _check_already_materializing() reads registry state, but MATERIALIZING is not written until the executor task starts. Two concurrent async=true requests can therefore both pass the check and enqueue duplicate materializations. Please atomically reserve the feature views before returning 202 (and roll the reservation back if submission fails), or use a registry compare-and-set/lock.
MaterializeIncrementalRequest.version is accepted and sent by the SDK, but both server paths call store.materialize_incremental() without version=request.version; authorization resolution also ignores it. A versioned request can therefore materialize a different feature-view version than requested. Please thread version through both sync and async paths and cover it in the server tests.

The existing review thread about the SDK no longer exposing synchronous mode also remains unresolved.

jyejare

This PR implements remote materialization support, allowing clients with a remote online store topology to delegate materialization requests to a feature server via async HTTP endpoints. The implementation adds proper conflict detection, state management, and force override capabilities.

jyejare · 2026-07-29T15:28:58Z


        Rolls back all already-transitioned FVs if this one can't transition.
        """
-        previous_state = getattr(feature_view, "state", None)
+        previous_states[feature_view.name] = getattr(feature_view, "state", None)


[Critical] Race condition in state tracking initialization

The line previous_states[feature_view.name] = getattr(feature_view, "state", None) was moved before the state transition logic, but this creates a critical bug. If the transition fails and we need to rollback other FVs, the previous_states dict won't have entries for FVs that failed before this one was reached.

Suggested:

Suggested change

Rolls back all already-transitioned FVs if this one can't transition.

"""

previous_state = getattr(feature_view, "state", None)

previous_states[feature_view.name] = getattr(feature_view, "state", None)

previous_state = getattr(feature_view, "state", None)

if (

hasattr(feature_view, "state")

and feature_view.state != FeatureViewState.STATE_UNSPECIFIED

):

# ... validation logic ...

feature_view.state = FeatureViewState.MATERIALIZING

self.registry.apply_feature_view(feature_view, self.project, commit=True)

previous_states[feature_view.name] = previous_state

Thanks for the careful look — walked through both placements with the caller loop in mind; they have the same observable outcome (including failure cases).
Caller (simplified):

for feature_view, fv_start in fv_with_dates: self._transition_fv_to_materializing(feature_view, regular_fvs, previous_states) regular_fvs.append(feature_view) # only after a successful return

On a failed transition, the failing FV is not in already_transitioned / regular_fvs. Rollback only restores FVs that already succeeded.

jyejare · 2026-07-29T15:28:58Z

+                # by store.materialize(); server only accepts and runs in background.
+                def _run_materialize():
+                    try:
+                        store.materialize(
+                            start_date,
+                            end_date,
+                            fv_names,
+                            disable_event_timestamp=request.disable_event_timestamp,
+                            full_feature_names=request.full_feature_names,
+                            version=request.version,
+                        )
+                    except Exception as e:
+                        logger.error(
+                            f"Async materialization failed for {fv_names}: {e}",


[Critical] Background task exception handling insufficient

The async materialization runs in a background thread but doesn't handle critical failures properly. If materialization fails, only the FV state is reset to GENERATED, but there's no way for the client to know the operation failed since the API already returned 202. This could lead to silent failures in production.

Suggested:

Suggested change

# by store.materialize(); server only accepts and runs in background.

def _run_materialize():

try:

store.materialize(

start_date,

end_date,

fv_names,

disable_event_timestamp=request.disable_event_timestamp,

full_feature_names=request.full_feature_names,

version=request.version,

)

except Exception as e:

logger.error(

f"Async materialization failed for {fv_names}: {e}",

except Exception as e:

logger.error(

f"Async materialization failed for {fv_names}: {e}",

exc_info=True,

)

_update_fv_state(store, fv_names, FeatureViewState.GENERATED)

# TODO: Consider implementing a status endpoint or webhook callback

# for clients to check materialization status

Thanks — agreed that fire-and-forget 202 means clients don’t get the failure on the HTTP response. That’s intentional for this PR: async acceptance + observe completion via FeatureView.state / materialization_intervals (and run_async=False when the client needs the sync HTTP path to block/error).

A dedicated status endpoint / job handle / webhook is out of scope here and planned as a follow-up (along the RemoteComputeEngine / job-tracking direction discussed earlier). I’ve added a TODO in the failure path comment pointing to that follow-up.

jyejare · 2026-07-29T15:28:58Z

+            content={
+                "error": (
+                    f"Cannot start async materialization — the following feature "
+                    f"views are already in MATERIALIZING state: {conflicting}. "
+                    f"Use ?force=true to override."
+                ),
+                "feature_views": conflicting,
+            },
+        )
+    return None
+
+
+def _update_fv_state(


[Warning] Silent exception handling could hide real errors

The _reset_stuck_materializing_to_generated and _check_already_materializing functions silently ignore all exceptions when accessing feature views. This could hide legitimate errors like registry corruption or network issues, making debugging difficult.

Suggested:

Suggested change

content={

"error": (

f"Cannot start async materialization — the following feature "

f"views are already in MATERIALIZING state: {conflicting}. "

f"Use ?force=true to override."

),

"feature_views": conflicting,

},

)

return None

def _update_fv_state(

except (FeatureViewNotFoundException, KeyError):

# Expected when FV doesn't exist

pass

except Exception as e:

logger.warning(f"Unexpected error checking state for {fv_name}: {e}")

pass

jyejare · 2026-07-29T15:28:58Z

            or normalize_version_string(self.version)
            != normalize_version_string(other.version)
            or self.org != other.org
+            or self.state != other.state


[Suggestion] State comparison added to eq affects hash consistency

Adding state to the equality comparison means two otherwise identical FeatureViews will be considered different if they're in different states (e.g., GENERATED vs MATERIALIZING). This could break set operations, dict lookups, and caching logic that expects state changes to not affect object identity.

Suggested:

Suggested change

or self.state != other.state

# Consider if state should be included in equality. If FeatureViews should be

# considered equal regardless of materialization state, remove this line.

# If state is semantically important for equality, ensure __hash__ is also updated

# or make the class unhashable.

jyejare · 2026-07-29T15:28:58Z

    feature_views: Optional[List[str]] = None
    disable_event_timestamp: bool = False
    full_feature_names: bool = False
+    version: Optional[str] = None


 class MaterializeIncrementalRequest(BaseModel):
    end_ts: str
    feature_views: Optional[List[str]] = None
    full_feature_names: bool = False
+    version: Optional[str] = None


 class GetOnlineFeaturesRequest(BaseModel):


[Suggestion] Missing version parameter documentation and validation

The version parameter was added to both MaterializeRequest and MaterializeIncrementalRequest but there's no documentation about what values are valid or how it affects the materialization behavior.

Suggested:

Suggested change

feature_views: Optional[List[str]] = None

disable_event_timestamp: bool = False

full_feature_names: bool = False

version: Optional[str] = None

class MaterializeIncrementalRequest(BaseModel):

end_ts: str

feature_views: Optional[List[str]] = None

full_feature_names: bool = False

version: Optional[str] = None

class GetOnlineFeaturesRequest(BaseModel):

version: Optional[str] = Field(

None,

description="Optional version to materialize (e.g., 'v2'). Requires feature_views with exactly one entry."

)

jyejare · 2026-07-29T15:28:58Z

@@ -839,27 +971,49 @@ async def materialize(request: MaterializeRequest) -> None:
            )



[Nitpick] Inconsistent parameter passing in sync mode

In the synchronous code path for materialize, the version parameter is passed but the feature_views parameter uses the original request.feature_views instead of the resolved fv_names list.

Suggested:

Suggested change

await run_in_threadpool(

store.materialize,

start_date,

end_date,

fv_names, # Use resolved names for consistency

disable_event_timestamp=request.disable_event_timestamp,

full_feature_names=request.full_feature_names,

version=request.version, # Don't forget version parameter

)

sync path now passes resolved fv_names and version=request.version (aligned with async).

… threading - Revert FeatureView.state from __eq__ - Add run_async for remote sync vs async HTTP - Reserve MATERIALIZING before 202; idempotent store transitions - Thread version through authorize and all materialize server paths - Narrow silent excepts; document version on request models Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

aniketpalu · 2026-07-31T11:00:18Z

@franciscojavierarceo

Thanks — both blocking items addressed in d8e7a99:

Race / reserve before 202: async path now sets MATERIALIZING before returning 202. Concurrent requests hit 409. store.materialize() treats already-MATERIALIZING as a no-op (idempotent transition; rollback target GENERATED for the reserved case).
version threading: version is passed through _authorize_materialize_views(...) and into both sync and async calls for /materialize and /materialize-incremental.
Also resolved the SDK sync gap via run_async (see related thread).
Still TODO on my side: add server unit coverage that asserts version is forwarded on both paths — will follow up shortly.

ntkathole · 2026-07-31T12:21:02Z

+                        _update_fv_state(store, fv_names, FeatureViewState.GENERATED)
+
+                loop = asyncio.get_running_loop()
+                loop.run_in_executor(None, _run_materialize_incremental)


non-blocking but I think it's better if we used dedicated executor, instead of default shared pool, for materialization so that it won't block other operations if multiple executors in progress

ntkathole · 2026-07-31T12:30:04Z

+        endpoint: str,
+        payload: Dict[str, Any],
+        force: bool = False,
+        run_async: bool = True,


should this be by default False ? since it's change in behavior for existing users - the call returns successfully even if the server-side materialization fails later.

Agree, updating soon

…k-e2e

aniketpalu requested a review from a team as a code owner July 28, 2026 11:11

aniketpalu and others added 2 commits July 28, 2026 16:43

Merge branch 'master' into feat/remote-materialize-v2

9e121fc

aniketpalu added 3 commits July 28, 2026 17:43

fix: Conflicts resolved

9a97b70

Signed-off-by: Aniket Paluskar <apaluska@redhat.com>

aniketpalu mentioned this pull request Jul 29, 2026

feat: Add remote materialization support via feature server #6590

Closed

7 tasks

ntkathole added the ok-to-test label Jul 29, 2026

ntkathole reviewed Jul 29, 2026

View reviewed changes

Comment thread sdk/python/feast/feature_view.py Outdated

ntkathole reviewed Jul 29, 2026

View reviewed changes

franciscojavierarceo requested changes Jul 29, 2026

View reviewed changes

jyejare requested changes Jul 29, 2026

View reviewed changes

aniketpalu and others added 2 commits July 30, 2026 14:43

Merge branch 'master' into feat/remote-materialize-v2

dc6f965

aniketpalu requested review from franciscojavierarceo, jyejare and ntkathole July 31, 2026 12:12

ntkathole reviewed Jul 31, 2026

View reviewed changes

aniketpalu added a commit to aniketpalu/feast that referenced this pull request Jul 31, 2026

Merge PR feast-dev#6649 (feat/remote-materialize-v2) into merged-spar…

0ea48e8

…k-e2e

Merge branch 'master' into feat/remote-materialize-v2

059c2d2

-        Rolls back all already-transitioned FVs if this one can't transition.
-        """
-        previous_state = getattr(feature_view, "state", None)
-        previous_states[feature_view.name] = getattr(feature_view, "state", None)
+        previous_state = getattr(feature_view, "state", None)
+        if (
+            hasattr(feature_view, "state")
+            and feature_view.state != FeatureViewState.STATE_UNSPECIFIED
+        ):
+            # ... validation logic ...
+            feature_view.state = FeatureViewState.MATERIALIZING
+            self.registry.apply_feature_view(feature_view, self.project, commit=True)
+        previous_states[feature_view.name] = previous_state

-            or self.state != other.state
+# Consider if state should be included in equality. If FeatureViews should be
+# considered equal regardless of materialization state, remove this line.
+# If state is semantically important for equality, ensure __hash__ is also updated
+# or make the class unhashable.

		@@ -839,27 +971,49 @@ async def materialize(request: MaterializeRequest) -> None:
		)

Uh oh!

Conversation

aniketpalu commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Client-side (SDK)

Server-side

Registry

Which issue(s) this PR fixes

Checks

Testing Strategy

Misc

Uh oh!

codecov-commenter commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franciscojavierarceo left a comment

Choose a reason for hiding this comment

Uh oh!

jyejare left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aniketpalu Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aniketpalu commented Jul 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aniketpalu commented Jul 28, 2026 •

edited

Loading

codecov-commenter commented Jul 28, 2026 •

edited

Loading

aniketpalu Jul 31, 2026 •

edited

Loading