[fix](arrow-flight) Support table type metadata and filters - #67899
hutiefang76 wants to merge 2 commits into
Conversation
### What problem does this PR solve? Related to apache#67578 (partial A3). Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows. Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends. ### Release note Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately. ### Check List (For Author) - Test: - [x] Unit Test: 30 passed via run-fe-ut.sh, including 19 new regression cases and 11 existing producer/schema cases. - Baseline: the 19 new cases produced 15 failures/errors on unmodified master, reproducing UNIMPLEMENTED and incorrect filtering. - Coverage includes both schema modes, filter order, unknown/duplicate types, external catalog propagation, serialized column alignment and repeated success/failure buffer cleanup. - FE Checkstyle: 0 violations. - No live-cluster or end-to-end ADBC driver test was run. - Behavior changed: Yes. GetTableTypes now returns BASE TABLE, SYSTEM VIEW and VIEW. Empty type filters retain all authorized tables; unknown-only filters return no rows. - Does this need documentation: No new configuration or API schema. Validation command (JDK 17, matching Thrift 0.24.0 compiler): ```sh bash run-fe-ut.sh --run org.apache.doris.arrowflight.FlightSqlTableTypesTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperTableTypesTest,org.apache.doris.arrowflight.DorisFlightSqlProducerTest,org.apache.doris.arrowflight.FlightSqlSchemaHelperArrowTypeTest ```
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at |
There was a problem hiding this comment.
Could you please also add a sql based regression test using arrow flight ?
There was a problem hiding this comment.
Added in 4937418.
Added test_table_types_over_arrow_flight under arrow_flight_sql_p0. It creates a base table and a view over Arrow Flight SQL, then consumes DatabaseMetaData.getTableTypes() and getTables() through the Flight JDBC driver. The checks cover unfiltered requests, individual types, both multi-type orders, and unknown/duplicate types, including catalog/schema values.
Validation: the official regression runner passed the new suite (1 run, 0 failures/skips). Locally I used Doris's test FE with real MySQL/Flight services and its built-in mocked BE; this exercises SQL DDL and the metadata RPCs, but is not a native-BE/data-scan integration run. The 30 existing focused FE tests also pass.
bash run-regression-test.sh --run -d arrow_flight_sql_p0 -s test_table_types_over_arrow_flight -g arrow_flight_sql### What problem does this PR solve? Related PR: apache#67899 Problem Summary: Exercise SQL-created base tables and views through the Arrow Flight JDBC metadata APIs, covering type enumeration and complete type filters. The suite executes DDL over Flight and consumes GetTableTypes/GetTables streams. ### Release note None ### Check List (For Author) - Test: official regression runner, 1 suite passed against a test FE with real Flight/MySQL services and the built-in mocked BE; 30 related FE unit tests passed. No native BE or data-scan integration was exercised. - Behavior changed: No, test-only addition - Does this need documentation: No
What problem does this PR solve?
Related to #67578 (partial A3).
Flight SQL clients can obtain FlightInfo for GetTableTypes, but reading its ticket fails with UNIMPLEMENTED. GetTables also forwards only the first table_types entry to a service that treats only VIEW as a filter: [VIEW, BASE TABLE] loses base tables, while [BASE TABLE] or an unknown type can return excluded rows.
Implement the type stream using the same Doris names as GetTables, sorted according to Flight SQL. Match the complete filter against the existing authorized listing before collecting optional schemas. Request only name/type metadata to avoid unused table statistics, and retain the session guard and Arrow buffer cleanup on failed sends.
Release note
Flight SQL clients can enumerate supported table types and filter table metadata by multiple types accurately.
Check List (For Author)
test_table_types_over_arrow_flight, 1 suite passed via the official regression runner. SQL DDL and metadata RPCs ran against a real test FE with the built-in mocked BE.Validation command (JDK 17, matching Thrift 0.24.0 compiler):