Support mapping canonical extension types to Parquet LogicalTypes

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• @mbrobbel added initial support for arrow extension types in Add ExtensionType trait and CanonicalExtensionType enum #5822

However, these extension type annotations apparently do not survive round trip write/reading to parquet as described in this comment with @kylebarron: https://github.com/apache/arrow-rs/pull/5822/files#r1925627198

Describe the solution you'd like
Make extension tests work round trip through parquet

Describe alternatives you've considered

Additional context

[alamb]

@mbrobbel and @kylebarron -- this item was brought up as something required to implement Varaint support (#6736). Do you by any chance have any additional information on what is required / how it might be implemented you can share (so we can make progress)?

cc @PinkCrow007

[paleolimbot]

We'll likely need this for Geometry/Geography as well! I can look more closely if I'm the one to start in on that change, but in C++ we handle something like that with an option to the reader properties:

https://github.com/apache/arrow/blob/c56ec12a2f38ed6fce2d87ea65345b6edf4234f5/cpp/src/parquet/properties.h#L988-L998

...although something more like a registry might be more flexible (e.g., so that a type could be registered with, for example, a DataFusion context like a UDT might be registered in Spark or DuckDB). (With apologies if I missed the major API challenge here 😬 )

[mbrobbel]

@mbrobbel and @kylebarron -- this item was brought up as something required to implement Varaint support (#6736). Do you by any chance have any additional information on what is required / how it might be implemented you can share (so we can make progress)?

cc @PinkCrow007

My thinking was: given an Arrow field with an extension type and writing this field to Parquet and then reading it again (using the provied Arrow writers/readers in the parquet crate) should retain the extension type information i.e. after reading this field (from Parquet) it should still have the same extension type.

From briefly looking at this I think we can implement this by replacing some parquet impls to operate on fields instead of types.

[paleolimbot]

replacing some parquet impls to operate on fields instead of types.

I'm experimenting with this in DataFusion, too ( apache/datafusion#15036 ) and there it's a fairly disruptive change (particularly for the UDFs). I know the version of DataType::Extension(name, metadata) was rejected but I wonder if DataType::Extension(extension_type) (never created without opt-in) would help minimize the surface area of supporting these types. (Although here there are probably sufficiently few internals that the change is not that difficult).

[alamb]

replacing some parquet impls to operate on fields instead of types.

I'm experimenting with this in DataFusion, too ( apache/datafusion#15036 ) and there it's a fairly disruptive change (particularly for the UDFs). I know the version of DataType::Extension(name, metadata) was rejected but I wonder if DataType::Extension(extension_type) (never created without opt-in) would help minimize the surface area of supporting these types. (Although here there are probably sufficiently few internals that the change is not that difficult).

I would personally be very open to the idea of adding a new enum to DataType. While it may not be strictly necessary from an API point of view, if it reduces churn and makes it easier to roll out extension types across the ecosystem it makes a lot of sense to me

We could also potentially feature flag it for a while (feature=extension) as a way to roll it out before the next major release 🤔

[mbrobbel]

The spec defines extension types as metadata on Fields, so I think adding an Extension variant to DataType is going to cause confusing situations. But maybe I'm misunderstanding what you are suggesting?

[paleolimbot]

The spec defines extension types as metadata on Fields

I believe the language in the spec is limited to exchange via FFI and serialization via IPC (e.g., Java, Go, and Arrow C++ include extension types as objects with native fields and only serialize when specifically requested). I don't believe it has caused confusing situations in other languages but it could be worth asking others! I will give it a try anyway and see what kind of implementation effort/downstream impacts it may have.

[mbrobbel]

The spec defines extension types as metadata on Fields

I believe the language in the spec is limited to exchange via FFI and serialization via IPC (e.g., Java, Go, and Arrow C++ include extension types as objects with native fields and only serialize when specifically requested). I don't believe it has caused confusing situations in other languages but it could be worth asking others! I will give it a try anyway and see what kind of implementation effort/downstream impacts it may have.

The DataType approach requires all matches on instances of DataType to consider the extension type variant inner datatype, which has major downstream impact. It means that consumers that don't care about extension types now need to deal with them. The spec states that implementations are not required to support canonical extension types.