Skip to content

Pipe: Mask sensitive attributes in sink subtask display strings#17737

Open
luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:fix/pipe-sink-mask-sensitive-attributes
Open

Pipe: Mask sensitive attributes in sink subtask display strings#17737
luoluoyuyu wants to merge 1 commit into
apache:masterfrom
luoluoyuyu:fix/pipe-sink-mask-sensitive-attributes

Conversation

@luoluoyuyu
Copy link
Copy Markdown
Member

Description

Use masked PipeParameters display string for logs, metrics and subtask names while keeping unmasked sorted string for internal lifecycle map keys. Also treat scp.password as a sensitive parameter.


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

Use masked PipeParameters display string for logs, metrics and subtask
names while keeping unmasked sorted string for internal lifecycle map keys.
Also treat scp.password as a sensitive parameter.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Caideyipi
Copy link
Copy Markdown
Collaborator

Medium: iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/subtask/sink/PipeSinkSubtaskManager.java:123 now writes
the masked display string into the sink runtime environment and subtask ID/tag path, but iotdb-core/datanode/src/main/java/org/
apache/iotdb/db/pipe/metric/sink/PipeDataRegionSinkMetrics.java:200 still keys compressionTimerMap only by
sink.getAttributeSortedString(), and the sinks look timers up only by that same string (iotdb-core/datanode/src/main/java/org/apache/
iotdb/db/pipe/sink/protocol/thrift/async/IoTDBDataRegionAsyncSink.java:512, iotdb-core/datanode/src/main/java/org/apache/iotdb/db/
pipe/sink/protocol/thrift/sync/IoTDBDataRegionSyncSink.java:596, iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/
protocol/airgap/IoTDBDataRegionAirGapSink.java:604). That means two active sinks that differ only in masked fields like password,
scp.password, or ssl.trust-store-pwd will create separate subtasks but share one compression-timer entry; when either subtask is
removed, iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/sink/PipeDataRegionSinkMetrics.java:389 removes that
shared key. Result: PIPE_COMPRESSION_TIME attribution becomes wrong, and a live sink can lose its timer mapping.

  • Low: iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/subtask/sink/PipeSinkSubtaskManager.java:200 still returns
    the raw canonical attribute string, and its failure paths still append that raw string directly into exceptions (iotdb-core/datanode/
    src/main/java/org/apache/iotdb/db/pipe/agent/task/subtask/sink/PipeSinkSubtaskManager.java:209, iotdb-core/datanode/src/main/java/
    org/apache/iotdb/db/pipe/agent/task/subtask/sink/PipeSinkSubtaskManager.java:237, iotdb-core/datanode/src/main/java/org/apache/iotdb/
    db/pipe/agent/task/subtask/sink/PipeSinkSubtaskManager.java:261). So if those exceptions are logged, the secret is still exposed on
    error paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants