Skip to content

StreamingTable does not project physical sort expressions #19717

@timsaucer

Description

@timsaucer

Describe the bug

When performing a scan using a StreamingTable that contains sort expressions, if the projection does not include one of the sort columns it leads to an error.

PhysicalExpr Column references column 'b' at index 1 (zero-based) but input schema only has 1 columns: [\"a\"]

To Reproduce


#[tokio::test]
async fn test_sort_with_streaming_table() -> Result<()> {
    let batch = record_batch!(
            ("a", Int32, [1, 2, 3]),
            ("b", Int32, [1, 2, 3])
    )?;

    let ctx = SessionContext::new();

    let sort_order = vec![
        SortExpr::new(
            Expr::Column(datafusion_common::Column::new(
                Option::<TableReference>::None,
                "a",
            )),
            true,
            false,
        ),
        SortExpr::new(
            Expr::Column(datafusion_common::Column::new(
                Option::<TableReference>::None,
                "b",
            )),
            true,
            false,
        ),
    ];
    let schema = batch.schema();
    let batches = Arc::new(DummyStreamPartition {
        schema: schema.clone(),
        batches: vec![batch],
    }) as _;
    let provider = StreamingTable::try_new(schema.clone(), vec![batches])?
        .with_sort_order(sort_order);
    ctx.register_table("test_table", Arc::new(provider))?;

    let sql = "SELECT a FROM test_table group by a";
    ctx.sql(sql).await?.show().await?;

    Ok(())
}

Expected behavior

The above test should pass.

Additional context

I think this can probably be resolved in StreamingTable::scan. We can project the schema before calling create_physical_sort_exprs

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions