Investigate the relation between the visibility, weight and flag arrays

[sjperkins]

In katdal, there is interplay between the values in the above 3 arrays in the sense that, for example, the final visibility values presented to the developer might be dependent on the weights and flags (and vice versa).

This is important to know about since we wish to avoid multiple network requests for the same data to the archive. In katdal this is achieved by using dask arrays which are backed by a directed acyclic graph (DAG). By contrast, xarray-kat uses tensorstore which doesn't quite have this level of expressiveness: the strategy for avoiding repeated data requests is to use caches, which can be associated with different tensorstores which can be layered ontop of each other.

The fortunate part of this is that it sounds from conversations with @ludwigschwardt that this happens chunk-wise: there isn't cross-communication between chunks of different arrays.

However, it would be good to get a sense of exactly how the visibility, weight and flag arrays interact with each other before they are presented to the developer. @ludwigschwardt is this documented anywhere? If not could you please reply to this issue with a mermaid diagram? Or an image with a hand-drawn diagram would do.

Then, it would probably be possible to code up the xarray BackendArray's that present the final data to the user. For example, WeightArray combines both integer and channel weights to provide full resolution weights.

xarray-kat/src/xarray_kat/array.py

Lines 146 to 189 in 38dd59d

	class WeightArray(BackendArray, CorrProductMixin):
	"""Combines weights and channel_weights to present a unified WEIGHT array"""
	_int_weight_prov: MeerkatStoreProvider
	_channel_weight_prov: MeerkatStoreProvider
	_cp_argsort: npt.NDArray
	_npol: int
	def __init__(
	self,
	int_weight_prov: MeerkatStoreProvider,
	channel_weight_prov: MeerkatStoreProvider,
	cp_argsort: npt.NDArray,
	npol: int,
	):
	# Integer weights have a (time, frequency, corrprod) shape
	CorrProductMixin.__init__(self, int_weight_prov.store.shape, cp_argsort, npol)
	self._int_weight_prov = int_weight_prov
	self._channel_weight_prov = channel_weight_prov
	self._cp_argsort = cp_argsort
	self._npol = npol
	@property
	def dtype(self) -> npt.DTypeLike:
	return np.dtype(self._channel_weight_prov.store.dtype.type)
	def __getitem__(self, key) -> npt.NDArray:
	return explicit_indexing_adapter(
	key, self.shape, IndexingSupport.OUTER, self._getitem
	)
	def _getitem(self, key) -> npt.NDArray:
	corrprod_key = self.meerkat_key(key)
	chan_weight_key = (corrprod_key[0], None, corrprod_key[1], None)
	chan_weight_store = self._channel_weight_prov.store[chan_weight_key]
	int_weight_store = ts.cast(
	self._int_weight_prov.store[corrprod_key].transpose(self.transpose_axes),
	chan_weight_store.dtype,
	)
	# Issue reads at the same time, then await their completion
	int_weight_fut = int_weight_store.read()
	chan_weight_fut = chan_weight_store.read()
	return int_weight_fut.result() * chan_weight_fut.result()

If necessary a visibility_store_provider can be added that can be used to modify the weights, for example.

[sjperkins]

However, it would be good to get a sense of exactly how the visibility, weight and flag arrays interact with each other before they are presented to the developer.

The following diagram illustrate this relation