Support imperfect dataframe in `set_data()` calls

[briochh]

Not really an issue. Just wondering about how straightforward it would be to support passing imperfect dataframes in set_data() assignments?

Currently flopy is tolerant of recarrays being passed with excess data fields:

flopy/flopy/mf6/data/mfdataplist.py

Lines 670 to 686 in 9bc7aac

	if isinstance(data, np.recarray):
	# verify data shape of data (recarray)
	if len(data) == 0:
	# create empty dataset
	data = pandas.DataFrame(columns=self._header_names)
	elif len(data[0]) != len(self._header_names):
	if len(data[0]) == len(self._data_item_names):
	# data most likely being stored with cellids as tuples,
	# create a dataframe and untuple the cellids
	# In pandas 3+, DataFrame() with recarray requires columns to match
	# field names, so create without columns param then rename if needed
	data = pandas.DataFrame(data)
	if list(data.columns) != self._data_item_names:
	data.columns = self._data_item_names
	data = self._untuple_cellids(data)[0]
	# make sure columns are still in correct order
	data = pandas.DataFrame(data, columns=self._header_names)

However, if data is passed already as DataFrame it is expected to be perfect (with no additional columns) and with exactly the correct cellid column breakdown. [as an aside the error message below may not trigger because data[0] returns a KeyError first (iloc[0]?)]

flopy/flopy/mf6/data/mfdataplist.py

Lines 755 to 763 in 9bc7aac

	elif isinstance(data, pandas.DataFrame):
	if len(data.columns) != len(self._header_names):
	message = (
	f"ERROR: Data list {self._data_name} supplied the "
	f"wrong number of columns of data, expected "
	f"{len(self._data_item_names)} got {len(data[0])}.\n"
	f"Data columns supplied: {data.columns}\n"
	f"Data columns expected: {self._header_names}"
	)

Could the self._untuple_cellids(data)[0] step and extraction of self._header_names happen before the if len(data.columns) != len(self._header_names): check for DataFrame instances?

EDIT:

maybe additional fields aren't supported even when using recarrays (that might be nice)? But the recarray set_data() does support allow for a generic cellid field using that self._untuple_cellids(data)[0] step.

[wpbonelli]

Thanks for this @briochh. Got some fixes in for the key error and the recarray/dataframe cell ID tuple inconsistency.

maybe additional fields aren't supported even when using recarrays

Yeah, don't think we tolerate extra columns for either recarray or dataframe. There seems to be some precedent for that in other projects, e.g. polars has extra_fields but it defaults to "raise" . Maybe we keep it strict by default, and let the caller disable column checking, like .set_data(..., strict=False)? If so, should we warn when extra columns are dropped?

It wouldn't be hard to do this. Seems pretty low risk. But 4.x will not have .set_data(), instead will deal directly with dataframes and recarrays, so we'd have to decide whether tolerance becomes a higher-level option (component-level or global) if we want it. 4.x will make it much easier to see what the expected columns are, making it easier for calling code to drop extras, in any case. So maybe supporting imperfect tabular data becomes overkill.