DataFusion weekly project plan (Andrew Lamb) - April 1, 2024

@jayzhan211

Follow on to #9796

It would be great for other contributors to DataFusion who plan non trivial work could try to make them visible somehow as well 🙏 (feel free to copy / modify the format)

My Highlights from Last Week

37.0.0 release is looking 🚀 Release DataFusion 37.0.0 #9682
The community made great progress on unparsing expressions: Complete support for Expr --> String #9726
"extract scalar functions from the core" continues at good pace way [Epic] Port BuiltInFunctons to datafusion-functions-* crates #9285 (thanks @jayzhan211 and @erenavsarogullari and @Omega359 for keeping the 🚋 running here)
Submitted SIGMOD final draft : Finalize SIGMOD 2024 paper ~(if accepted)~ #8373
Ran our first in person meetup (it would be great if others wanted to organize meetups in other geographies): UPDATE: VENUE IS FULL Invitation: Mar 25, 2024 DataFusion meetup before the DataCouncil March 2024 in Austin TX #8522
We have submitted the proposal for a new top level project for DataFusion to the ASF board: [DISCUSS] Move Apache Arrow Datafusion to a new top level Apache project #6475 (comment)

My (personal) goals

1000 projects are built using DataFusion 📈

My (personal) plans for this week

Make at least two more PRs for subtasks [EPIC] Stop copying LogicalPlan during OptimizerPasses #9637
Catchup on review backlog
Writeup ideas on custom / user defined indexes (I mentioned this to @westonpace in person but haven't written it down yet)

Project Queue (list of future projects)

Projects I plan to help actively help review / plan

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can and do review and merge PRs, so this is not the priorities of the project as a whole, just the approximate algorithm I use to prioritize my own time

Priority:

Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
Improvements directly related to features needed for InfluxDB (my employer)
Documentation and test improvements (I view these as very strategically important)
PRs that I think are strategically important
Other new features / additions to functionality (note this is the lowest)

The top strategically important projects in my head are:

Anything that makes it easier to use DataFusion as a user (docs, examples, better APIs, etc), including Getting started guide for new users (who want to use DataFusion in their project)
Anything that improves DataFusion's quality such as bug fixes, and improved / expanded tests, etc
Making function packages modular and easier to mix/match: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation or the projects named above

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example, the CI passes, includes documentation, etc)

Background

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

Communicate / coordinate in the community
Help provide an interesting summary of what is happening in DataFusion this week

Really interested to see your thoughts on custom indices, its something were currently working on to improve performance of listing table.

Review queue

Arrow

DataFusion

Arrow:

feat: support reading and writingStringView and BinaryView in parquet (part 2) arrow-rs#5557
Support list_view_array & large_list_view layout and basic construction arrow-rs#5576

DataFusion

Sorry for the location and possibly dumb question, but shouldn't all functions be exported here: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/unicode/mod.rs#L138

I am probably using the library incorrectly but (for example) LENGTH will not be registered by the central context here: https://github.com/apache/arrow-datafusion/blob/2f550032140d42d1ee6d8ed86f7790766fa7302e/datafusion/core/src/execution/context/mod.rs#L1450

@seddonm1

Hi @seddonm1 👋

Sorry for the location and possibly dumb question, but shouldn't all functions be exported here: https://github.com/apache/arrow-datafusion/blob/main/datafusion/functions/src/unicode/mod.rs#L138

Yes that is the intention

I am probably using the library incorrectly but (for example) LENGTH will not be registered by the central context here:

https://github.com/apache/arrow-datafusion/blob/2f550032140d42d1ee6d8ed86f7790766fa7302e/datafusion/core/src/execution/context/mod.rs#L1450

I think length is registered as an alias of character_length

@alamb

Thanks @alamb . If you look at the first link I cannot see anywhere where the length alias is actually registered. My queries which previously worked started failing after this update. I haven't investigated if more functions have also not been registered.

aliases for character_length are defined in https://github.com/apache/arrow-datafusion/blob/4bd7c137e0e205140e273a7c25824c94b457c660/datafusion/functions/src/unicode/character_length.rs#L45 and is used during udf registration @ https://github.com/apache/arrow-datafusion/blob/4bd7c137e0e205140e273a7c25824c94b457c660/datafusion/core/src/execution/context/mod.rs#L2137

Arrow:

feat: support reading and writingStringView and BinaryView in parquet (part 2) arrow-rs#5557

DataFusion

DataFusion 37.0.0 is released: #9682 🌮

@Omega359

Thanks @Omega359 . I was thrown as the aliases are not registered here https://github.com/apache/arrow-datafusion/blob/3ae029988754c3fd3eb000abd4b76e643b9cbc7b/datafusion/execution/src/registry.rs#L174

@milenkovicm

BTW something cool for next week -- @milenkovicm WASM UDFs #9326 / #9326 (comment) / https://github.com/milenkovicm/wasaffi

@matthewmturner

Really interested to see your thoughts on custom indices, its something were currently working on to improve performance of listing table.

In case anyone else is following along, here are some items related to indexing:

While I had this on the brain, @matthewmturner I also filed #9964 as I suspect others will be interested in helping make ListingTable faster too

@jayzhan211

Reviews:
@jayzhan211 is starting to pull aggregate functions out of the core: #9960

Next week #10002

Navigation Menu

Repository navigation

DataFusion weekly project plan (Andrew Lamb) - April 1, 2024 #9899

My Highlights from Last Week

My (personal) goals

My (personal) plans for this week

Project Queue (list of future projects)

Projects I plan to help actively help review / plan

Algorithm for (my) prioritizing PR reviews

Background

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DataFusion weekly project plan (Andrew Lamb) - April 1, 2024 #9899

Description

My Highlights from Last Week

My (personal) goals

My (personal) plans for this week

Project Queue (list of future projects)

Projects I plan to help actively help review / plan

Algorithm for (my) prioritizing PR reviews

Background

Activity

matthewmturner commented on Apr 1, 2024

alamb commented on Apr 1, 2024

alamb commented on Apr 1, 2024

seddonm1 commented on Apr 3, 2024

alamb commented on Apr 3, 2024

seddonm1 commented on Apr 3, 2024

Omega359 commented on Apr 4, 2024

alamb commented on Apr 4, 2024

alamb commented on Apr 4, 2024

seddonm1 commented on Apr 4, 2024

alamb commented on Apr 5, 2024

alamb commented on Apr 5, 2024

alamb commented on Apr 5, 2024

alamb commented on Apr 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions