[Enhancement]Extend SimulcastEncoderAdapter to support adaptive streaming

[ipavlidakis]

Summary

This fixes quality scaler interference during simulcast transitions and ensures
the SimulcastEncoderAdapter (SEA) correctly reports the active encoder's runtime
adaptation hints when only one spatial layer is active.

Without this fix, the quality scaler remains active during simulcast and
continuously degrades the input resolution, causing all simulcast layer
dimensions to be derived from scaled-down frames (e.g. f=268x480 instead of
f=720x1280 for 720p capture). Additionally, the SEA was reporting aggregated
encoder info with scaling_settings = kOff even when only one layer was active,
preventing the quality scaler from starting in single-active-layer mode (1:1 calls).

Problem

Three interrelated issues combined to produce broken quality adaptation behavior
during simulcast transitions:

1. Quality scaler active during simulcast. When transitioning from a 1:1
   call to a group call (simulcast), the quality scaler was not disabled. It
   continued to scale down the input resolution, and since simulcast layer
   dimensions are computed from the input frame size, all layers (q, h, f) were
   recomputed from the degraded input. This produced cascading downgrades
   (quality:1 → quality:2 → quality:3 → quality:4) and eventually triggered
   simulcast layer count reduction from 3 to 2.

2. SEA reporting aggregated info in single-active-layer mode. The
   SimulcastEncoderAdapter only forwarded the active encoder's info when
   stream_contexts_.size() == 1. At runtime, SEA can have multiple stream
   contexts while only one layer is unpaused. In that state, SEA reported
   scaling_settings = kOff, so the quality scaler could never start — even in
   1:1 mode where it's needed.

3. Conservative min_pixels_per_frame floor for iOS encoders. The ObjC
   encoder bridge used the 2-argument ScalingSettings constructor, inheriting
   kDefaultMinPixelsPerFrame = 320*180 = 57600. This prevented the quality
   scaler from reducing resolution below ~180x320 (the quarter-resolution
   simulcast layer for 720p), limiting adaptation range on constrained networks.

Root Cause

Quality scaler during simulcast: ConfigureQualityScaler() in
VideoStreamEncoderResourceManager had no awareness of whether simulcast was
active. It would start/keep the quality scaler running whenever the encoder
reported QP thresholds or is_quality_scaling_allowed was set, regardless of
how many simulcast layers were active.

SEA encoder info: The bug is in SimulcastEncoderAdapter::GetEncoderInfo().
The old logic treated "single stream context" as equivalent to "single active
layer", but those are not the same thing at runtime. Once SEA enters
multi-encoder mode, the number of stream contexts stays greater than one even if
only one layer is currently unpaused.

FindRequiredActiveLayers: FindRequiredActiveLayers() in
encoder_stream_factory.cc returned the position of the first active layer
instead of the highest, which could discard upper active layers when lower ones
were inactive.

min_pixels_per_frame: The ObjC bridge in objc_video_encoder_factory.mm
used ScalingSettings(low, high) which defaults min_pixels_per_frame to
57600 — too conservative for iOS VideoToolbox encoders that can encode well
below that threshold.

Fix

1. Quality scaler suppression during simulcast

• Added simulcast_active_ flag to VideoStreamEncoderResourceManager.
• SetSimulcastActive(bool) is called from ReconfigureEncoder() based on the
  number of active simulcast layers (not configured streams — the config may
  always specify 3 streams while the SFU controls activation).
• ConfigureQualityScaler() now checks !simulcast_active_ as an additional
  condition for quality_scaling_allowed. When simulcast is active, the quality
  scaler is stopped and removed.

2. Simulcast transition handling in VideoStreamEncoder

• ReconfigureEncoder() now detects transitions between single-stream and
  simulcast based on num_active_layers > 1.
• Transition to simulcast (e.g. 3rd participant joins): disables the quality
  scaler and calls ResetAdaptationsForSimulcastChange() to clear accumulated
  resolution restrictions. The video source then delivers full-resolution frames
  for correct layer dimension computation.
• Transition from simulcast (e.g. 3rd participant leaves): re-enables the
  quality scaler from a clean state (zero downgrades).
• Added diagnostic logging at every transition point for operational debugging.

3. SEA single-active-layer encoder info forwarding

• When exactly one stream is unpaused, SEA now forwards the active encoder's
  runtime-sensitive fields: scaling_settings, supports_native_handle,
  has_trusted_rate_controller, is_hardware_accelerated, is_qp_trusted,
  resolution_bitrate_limits, min_qp, preferred_pixel_formats.
• SEA still preserves its aggregated simulcast fields: implementation_name,
  fps_allocation, requested_resolution_alignment,
  apply_alignment_to_all_simulcast_layers.
• When more layers become active again, SEA returns to aggregated behavior.

4. FindRequiredActiveLayers fix

• Changed FindRequiredActiveLayers() to return the position after the highest
  active layer (not the first), ensuring all active layers are preserved in the
  stream count.

5. Lower min_pixels_per_frame for iOS encoders

• Changed the ObjC encoder bridge to use ScalingSettings(low, high, 90*160)
  instead of ScalingSettings(low, high). This lowers the quality scaler floor
  from 57600 to 14400 pixels, allowing adaptation below the quarter-resolution
  simulcast layer on constrained networks (e.g. 3G).

Why This Is Safe

• The quality scaler suppression is strictly scoped to the multi-active-layer
  state. Single-active-layer behavior (1:1 calls) is unchanged.
• Restriction clearing on simulcast transition is self-correcting: even if the
  first frame after clearing is still degraded, the source delivers
  full-resolution frames on the next frame, triggering a reconfigure with
  correct dimensions.
• The SEA change does not affect init-time encoder selection, bypass mode,
  pre-init GetEncoderInfo(), bitrate allocation, or true multi-layer
  aggregation.
• The min_pixels_per_frame change only affects ObjC-bridged encoders (iOS
  VideoToolbox). The global kDefaultMinPixelsPerFrame remains unchanged for
  software and Android encoders.

Tests

Added regression coverage in:

• video_stream_encoder_unittest.cc: quality scaler restrictions are cleared
  when active layers increase from 1 to multiple, and from 2 to 3.
• simulcast_encoder_adapter_unittest.cc: forwarding active encoder info when
  only one layer is unpaused; restoring aggregated SEA behavior when a second
  layer becomes active.
• encoder_stream_factory_unittest.cc: stream count preservation for sparse
  active patterns, highest-only active, lowest-only active.

Validation

Validated with on-device iOS testing:

• 1:1 call on 3G: quality scaler correctly adapts resolution down and back up.
• 3rd participant joins: simulcast activates, quality scaler disables, layer
  dimensions match configured ratios from full capture resolution.
• 3rd participant leaves: quality scaler re-enables from clean state.
• Repeated transitions: no accumulated degradation across cycles.

Summary by CodeRabbit

• New Features

  • Added frame encryption and data packet cryptography support to iOS/macOS SDK
  • Added privacy manifest support for iOS/macOS frameworks
  • Added desktop capture support for macOS
  • Improved quality scaling behavior for simulcast streaming with runtime layer tracking

• Bug Fixes

  • Fixed audio recording to no longer stop when send streams are muted
  • Fixed encoder information to accurately reflect active spatial layers

• Refactor

  • Updated audio device module initialization API
  • Refined frame transformer attachment mechanisms
  • Improved quality scaler adaptation for multi-layer scenarios

[coderabbitai]

Caution

Review failed

Failed to post review comments

📝 Walkthrough

Walkthrough

This PR introduces simulcast layer adaptation for quality scaling, optimizes frame crypto buffer construction, forwards single-layer encoder characteristics through the simulcast adapter, adds desktop capture exports for macOS frameworks, updates iOS/macOS SDK wiring with privacy info and cryptor headers, and refactors test infrastructure with API signature and mock interface updates.

Changes

Simulcast Layer Adaptation and Quality Scaler Control

Layer / File(s)	Summary
Quality scaler adaptation control APIs video/adaptation/video_stream_encoder_resource_manager.h	VideoStreamEncoderResourceManager gains SetSimulcastActive() and ResetAdaptationsForSimulcastChange() public methods, plus simulcast_active_ state guarded by encoder_queue_.
Quality scaler gating on simulcast state video/adaptation/video_stream_encoder_resource_manager.cc	ConfigureQualityScaler() and ConfigureBandwidthQualityScaler() gate scaling on !simulcast_active_, and ResetAdaptationsForSimulcastChange() clears quality scaler restrictions during layer transitions.
VideoStreamEncoder simulcast state updates video/video_stream_encoder.h, video/video_stream_encoder.cc	VideoStreamEncoder tracks active simulcast layers via GetNumActiveSimulcastLayers() and UpdateSimulcastAdaptationState(), updating resource manager during encoder configuration and rate changes.
Highest-layer active stream discovery video/config/encoder_stream_factory.cc	FindRequiredActiveLayers() now returns highest active layer index + 1 instead of stopping at first active layer.
Simulcast layer adaptation unit tests video/config/encoder_stream_factory_unittest.cc, video/video_stream_encoder_unittest.cc	Unit tests verify quality scaler reset/creation, stream count reduction to include highest layer, sparse activation patterns, and active-layer tracking in fake encoders.

Single-Layer Encoder Info Forwarding

Layer / File(s)	Summary
GetEncoderInfo() single-layer info forwarding media/engine/simulcast_encoder_adapter.cc	SimulcastEncoderAdapter::GetEncoderInfo() tracks active_stream_count and forwards runtime-sensitive encoder fields (scaling, QP trust, native-handle support, min QP, resolution limits) from the sole active encoder when exactly one stream is unpaused.
MockVideoEncoder min_qp support media/engine/simulcast_encoder_adapter_unittest.cc	MockVideoEncoder gains set_min_qp() setter and min_qp_ member, with GetEncoderInfo() populating info.min_qp.
Simulcast adapter encoder info tests media/engine/simulcast_encoder_adapter_unittest.cc	Unit tests validate runtime-sensitive field forwarding in single-layer scenarios and aggregated behavior restoration when multiple layers unpause.

Frame Crypto Buffer Optimization and API Updates

Layer / File(s)	Summary
Buffer initialization optimization api/crypto/frame_crypto_transformer.cc	FrameCryptorTransformer::encryptFrame and decryptFrame construct frame_header, frame_trailer, payload, iv, and encrypted_buffer using CreateUninitializedWithSize() with explicit filling; DataPacketCryptor similarly uses CreateUninitializedWithSize(0) for data packet headers.
RTCFrameCryptor SetFrameTransformer attachment sdk/objc/api/peerconnection/RTCFrameCryptor.mm	RTCFrameCryptor sender and receiver initializers switch from encoder/depacketizer-specific transformer setters to SetFrameTransformer() on nativeRtpSender/nativeRtpReceiver, with reformatted constructor and accessor implementations.
RTCFrameCryptorKeyProvider scoped_refptr type update sdk/objc/api/peerconnection/RTCFrameCryptorKeyProvider.mm	nativeKeyProvider() return type changes from rtc::scoped_refptr to webrtc::scoped_refptr, with reflowed initializer and implementation formatting.
SDK build wiring for frame crypto sdk/BUILD.gn	Adds RTCDataPacketCryptor and RTCFrameCryptor headers/sources to peerconnectionfactory_base_objc, ../api/crypto:frame_crypto_transformer to deps, and exports cryptor headers in iOS/mac frameworks.

Desktop Capture Platform Abstraction and macOS Wiring

Layer / File(s)	Summary
ObjCDesktopCapturer rtc::Thread to webrtc::Thread migration sdk/objc/native/src/objc_desktop_capture.h, sdk/objc/native/src/objc_desktop_capture.mm	ObjCDesktopCapture changes thread_ pointee type from rtc::Thread to webrtc::Thread, with reformatted class declarations, constructor signatures, and implementation methods.
ObjCDesktopMediaList thread abstraction and formatting sdk/objc/native/src/objc_desktop_media_list.h, sdk/objc/native/src/objc_desktop_media_list.mm	ObjCDesktopMediaList changes thread_ from std::unique_ptr<rtc::Thread> to std::unique_ptr<webrtc::Thread>, with reformatted includes, class declarations, and implementation methods.
Desktop capture library target and macOS framework exports sdk/BUILD.gn	Introduces desktopcapture_objc library target, adds desktop capture headers to mac framework exports, and updates mac framework deps.

iOS and macOS SDK Framework Wiring

Layer / File(s)	Summary
Privacy info and audio renderer framework exports sdk/BUILD.gn	Adds darwin_privacy_info bundle target, RTCAudioRenderer.h to base_objc and framework exports, and extends native_api_audio_device_module to both iOS and macOS.
Framework bundle initialization and dependencies sdk/BUILD.gn	iOS/mac framework bundles add :darwin_privacy_info to deps, mac framework also includes :desktopcapture_objc, and metal_objc mac sources remove RTCMTLNSVideoView.m.

Test Infrastructure and Audio State Management

Layer / File(s)	Summary
MockTransformableVideoFrame time-based interface api/test/mock_transformable_video_frame.h	MockTransformableVideoFrame removes RTP header() accessor mock and adds GetPresentationTimestamp, ReceiveTime, CaptureTime, CanSetCaptureTime, SetCaptureTime, and SenderCaptureTimeOffset mock methods.
AudioState mute stream recording behavior audio/audio_state.cc, audio/audio_state_unittest.cc	AudioState::OnMuteStreamChanged() removes StopRecording() call on mute, preventing recording teardown on mute-only transitions, with test asserting no StopRecording when muted and StopRecording when stream removed.
TestAudioDeviceModule Environment parameter migration modules/audio_device/include/test_audio_device.cc	TestAudioDeviceModule::Create and TestAudioDeviceModuleImpl constructors change from TaskQueueFactory* to const Environment& env parameter.
OCMock teardown improvements and factory test refactoring sdk/objc/unittests/RTCCameraVideoCapturerTests.mm, sdk/objc/unittests/RTCPeerConnectionFactoryBuilderTest.mm	RTCCameraVideoCapturerTests adds stopMocking() calls; RTCPeerConnectionFactoryBuilderTest switches from OCMock strict mocks to Objective-C runtime swizzling for initializer interception.
ObjCVideoEncoder minimum pixels scaling floor sdk/objc/native/src/objc_video_encoder_factory.mm	ObjCVideoEncoder::GetEncoderInfo() adds kObjCEncoderMinPixelsPerFrame (90×160) constant and passes it to ScalingSettings when QP thresholds available.
iOS-only CreateMutedDetectAudioDeviceModule guard sdk/objc/native/api/audio_device_module.mm	CreateMutedDetectAudioDeviceModule is conditionally compiled only for WEBRTC_IOS via preprocessor guard.
WebRTC test build template improvements webrtc.gni	rtc_test template now ensures non-XCTest targets include //test:test_main in deps for non-Chromium builds, and apple_framework_bundle_with_umbrella_header narrows umbrella copy condition to is_mac only.

🎯 4 (Complex) | ⏱️ ~60 minutes

🐰 Hopping through the code like a vigilant architect,
These layers leap from single encoders to many streams so select,
Cryptographic buffers dance with threads new-threaded true,
While test mocks and frameworks adapt the view!
🐇✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 2.90% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title '[Enhancement]Extend SimulcastEncoderAdapter to support adaptive streaming' directly corresponds to the main changes in the PR, particularly the SimulcastEncoderAdapter modifications and simulcast layer adaptation logic.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch iliaspavlidakis/ios-1481-webrtcextend-simulcastencoderadapter-to-support-adaptive-m145

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}