Skip to content

Support a channel as a Sink #270

@V0ldek

Description

@V0ldek

Is your feature request related to a problem? Please describe.
There is interest in using the engine as a pull-based iterator (#243). While that is hard to achieve, we can have a partial solution of supporting a channel between threads as a Sink. A user can then spin the engine in a separate thread and then use the other end of the channel as a source of results from which to pull.

It might be beneficial to check if one of those impls should be used in rq itself. If the overhead is small, we might default to it on large files where writing the results to output may be a heavy operation. This might also be an option for online output.

Describe the solution you'd like
We should have two implementations.

  1. std::sync::mpsc, the stdlib implementation of a channel.
  2. crossbeam_channel, the most popular channel crate.

The other should be behind a feature flag to enable the optional dependency – we won't force users to pay for it if they don't need it.

Describe alternatives you've considered
We could implement Sink on other primitives, like crossbeam's ArrayQueue, but that seems too specific. People trying to get a lower-level primitive to work with the engine probably would be better suited by a custom Sink impl anyway.

Metadata

Metadata

Assignees

Labels

acceptance: go aheadReviewed, implementation can startarea: resultImprovements in query result reportingtype: featureNew feature or request

Type

No type

Projects

Status

Todo

Relationships

None yet

Development

No branches or pull requests

Issue actions