Skip to content

[fix](arrow-flight) Support table type metadata and filters - #67899

Open
hutiefang76 wants to merge 2 commits into
apache:masterfrom
hutiefang76:frank/doris-flight-table-types
Open

hutiefang76 wants to merge 2 commits into
apache:masterfrom
hutiefang76:frank/doris-flight-table-types

Conversation

@hutiefang76

@hutiefang76 hutiefang76 commented Sep 12, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Related to #67578 (partial A3).

Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows.

Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends.

Release note

Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately.

Check List (For Author)

  • Test:
    • Unit Test: 30 passed via run-fe-ut.sh, including 19 new regression cases and 11 existing producer/schema cases.
    • Baseline: the 19 new cases produced 15 failures/errors on unmodified master, reproducing UNIMPLEMENTED and incorrect filtering.
    • Coverage includes both schema modes, filter order, unknown/duplicate types, external catalog propagation, serialized column alignment and repeated success/failure buffer cleanup.
    • FE Checkstyle: 0 violations.
    • SQL/Flight JDBC regression: test_table_types_over_arrow_flight, 1 suite passed via the official regression runner. SQL DDL and metadata RPCs ran against a real test FE with the built-in mocked BE.
    • No native-BE/data-scan cluster or ADBC driver test was run.
  • Behavior changed: Yes. GetTableTypes now returns BASE TABLE, SYSTEM VIEW and VIEW. Empty type filters retain all authorized tables; unknown-only filters return no rows.
  • Does this need documentation: No new configuration or API schema.

Validation command (JDK 17, matching Thrift 0.24.0 compiler):

bash run-fe-ut.sh --run org.apache.doris.arrowflight.FlightSqlTableTypesTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperTableTypesTest,org.apache.doris.arrowflight.DorisFlightSqlProducerTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperArrowTypeTest

### What problem does this PR solve?

Related to apache#67578 (partial A3).

Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows.

Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends.

### Release note

Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately.

### Check List (For Author)

- Test:
  - [x] Unit Test: 30 passed via run-fe-ut.sh, including 19 new regression cases and 11 existing producer/schema cases.
  - Baseline: the 19 new cases produced 15 failures/errors on unmodified master, reproducing UNIMPLEMENTED and incorrect filtering.
  - Coverage includes both schema modes, filter order, unknown/duplicate types, external catalog propagation, serialized column alignment and repeated success/failure buffer cleanup.
  - FE Checkstyle: 0 violations.
  - No live-cluster or end-to-end ADBC driver test was run.
- Behavior changed: Yes. GetTableTypes now returns BASE TABLE, SYSTEM VIEW and VIEW. Empty type filters retain all authorized tables; unknown-only filters return no rows.
- Does this need documentation: No new configuration or API schema.

Validation command (JDK 17, matching Thrift 0.24.0 compiler):

```sh
bash run-fe-ut.sh --run org.apache.doris.arrowflight.FlightSqlTableTypesTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperTableTypesTest,org.apache.doris.arrowflight.DorisFlightSqlProducerTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperArrowTypeTest
```
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please also add a sql based regression test using arrow flight ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 4937418.

Added test_table_types_over_arrow_flight under arrow_flight_sql_p0. It creates a base table and a view over Arrow Flight SQL, then consumes DatabaseMetaData.getTableTypes() and getTables() through the Flight JDBC driver. The checks cover unfiltered requests, individual types, both multi-type orders, and unknown/duplicate types, including catalog/schema values.

Validation: the official regression runner passed the new suite (1 run, 0 failures/skips). Locally I used Doris's test FE with real MySQL/Flight services and its built-in mocked BE; this exercises SQL DDL and the metadata RPCs, but is not a native-BE/data-scan integration run. The 30 existing focused FE tests also pass.

bash run-regression-test.sh --run -d arrow_flight_sql_p0 -s test_table_types_over_arrow_flight -g arrow_flight_sql

### What problem does this PR solve?

Related PR: apache#67899

Problem Summary: Exercise SQL-created base tables and views through the Arrow
Flight JDBC metadata APIs, covering type enumeration and complete type filters.
The suite executes DDL over Flight and consumes GetTableTypes/GetTables streams.

### Release note

None

### Check List (For Author)

- Test: official regression runner, 1 suite passed against a test FE with real
  Flight/MySQL services and the built-in mocked BE; 30 related FE unit tests
  passed. No native BE or data-scan integration was exercised.
- Behavior changed: No, test-only addition
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants