feat: Implement resource requirements for components #194

Copilot · 2026-01-28T19:44:16Z

Summary

Adds resource specification support for components to enable Ray actor resource allocation and Tuner placement group optimization. Resources can be declared as class attributes on Component definitions (recommended) or overridden at instantiation time.

Changes

Core Schema

Added Resource Pydantic class in plugboard_schemas.component with fields: cpu, gpu, memory (integer bytes), resources (custom dict)
Supports full SI prefix notation:
- Decimal SI: n, u, m, k, M, G, T, P, E (e.g., "250m" → 0.25, "5k" → 5000)
- Binary: Ki, Mi, Gi, Ti, Pi, Ei (e.g., "10Mi" → 10,485,760 bytes)
Default: cpu=0.001, others zero
Implements to_ray_options() for Ray actor configuration
Memory stored as integer bytes for precision

Component Integration

Added resources class attribute to Component for declaring default resource requirements
Extended Component.__init__() with optional resources parameter to override class-level defaults
Updated ComponentArgsDict, ComponentArgsSpec for YAML support
Resources auto-exported and reconstructed from configuration
Each component instance creates an instance-scoped copy of resources

Ray Execution

Modified RayProcess._create_component_actor() to apply resource requirements via ray.remote(**ray_options)
Falls back to default Resource() when unspecified
Import statements organized at top of file

Tuner Optimization

Added Tuner._calculate_placement_bundles() to aggregate component resources
Creates PlacementGroupFactory with tuner overhead (0.5 CPU) + component bundle
Import statements organized at top of file

Documentation

Updated Component class docstring to document the resources class attribute
Added Component.__init__() docstring explaining resource parameter behavior
Added comprehensive section in docs/examples/tutorials/running-in-parallel.md explaining both approaches:
- Class-level resource declaration (recommended)
- Constructor override for flexibility
Python and YAML examples demonstrating both resource specification approaches

Usage Examples

Class-level declaration (recommended):

from plugboard.component import Component
from plugboard.schemas import Resource

class CPUIntensiveTask(Component):
    io = IO(inputs=["x"], outputs=["y"])
    resources = Resource(cpu=2.0)  # Declare at class level
    
    async def step(self) -> None:
        # Your logic here
        pass

# Uses class-level resources
component = CPUIntensiveTask(name="my-task")

Constructor override:

# Override class-level resources when needed
component = CPUIntensiveTask(
    name="my-task",
    resources=Resource(cpu=4.0)  # Override to 4 CPUs
)

YAML config:

components:
  - type: my.CPUIntensiveTask
    args:
      name: my-task
      # Optionally override class-level resources
      resources:
        cpu: 4.0
        memory: "512Mi"

Tests

Unit tests: Resource parsing with all SI prefixes, validation, Ray conversion
Integration tests: Component/ProcessSpec/Tuner scenarios, class-level resource declaration
Manual verification (full test suite requires Ray dependencies)

Original prompt

This section details on the original issue you should resolve

<issue_title>feat: Implement resource requirements for components</issue_title>
<issue_description>### Summary

Users would like to be able to specify custom resource requirements on Component objects. These can be passed to Ray when a RayProcess is used or a Tuner object is run.

Requirements:

Must be able to specify Resource requirements for cpu, gpu, memory, resources. These should be defined as a Pydantic class in plugboard_schemas.

Must be able to specify resources as a numerical value, or as a string (to be validated by Pydantic), e.g. 250m for 0.25 or 10Mi for 1024 * 1024 * 10. resources should be a dictionary of string key, numerical value pairs.

Must be able to convert these requirements to their Ray equivalents, see https://docs.ray.io/en/latest/ray-core/api/doc/ray.actor.ActorClass.options.html, i.e. cpu = "250m" gets converted to num_cpus=0.25.

Must be able to pass a Resource to a Component when instantiating it. This will require changes to the Component and its schema, so that we can specify resource requirements in YAML config.

The default requirement should be {"cpu": 0.001}, to be used if the user does not provide anything.

Resource requirements should be passed to the actor options in RayProcess.

They should be used in the resource placement group inside Tuner.

Example

No response

Alternatives

No response</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes feat: Implement resource requirements for components #193

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>

plugboard-schemas/plugboard_schemas/component.py

plugboard/process/ray_process.py

examples/tutorials/004_using_ray/resources-example.yaml

plugboard/tune/tune.py

toby-coleman

@copilot see comments

…r, move imports to top Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>

…ource-requirements

github-actions · 2026-02-07T18:32:47Z

Benchmark comparison for 140dccf7 (base) vs 3301419f (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests ------------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     448.3365 (1.0)      465.6030 (1.01)     457.8813 (1.0)      6.9427 (4.07)     457.8773 (1.0)      11.0587 (4.78)          2;0  2.1840 (1.0)           5           1
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         457.2382 (1.02)     461.4251 (1.0)      458.7039 (1.00)     1.7056 (1.0)      457.9881 (1.00)      2.3129 (1.0)           1;0  2.1801 (1.00)          5           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

github-actions · 2026-02-07T18:35:48Z

Benchmark comparison for 140dccf7 (base) vs 9b361df5 (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     445.6866 (1.0)      452.1979 (1.0)      449.3371 (1.0)      2.5165 (1.0)      449.3286 (1.0)      3.6126 (1.0)           2;0  2.2255 (1.0)           5           1
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         446.5589 (1.00)     457.6603 (1.01)     452.1729 (1.01)     4.8984 (1.95)     453.9259 (1.01)     8.5801 (2.38)          2;0  2.2115 (0.99)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

github-actions · 2026-02-07T18:44:44Z

Benchmark comparison for 140dccf7 (base) vs d7c87c6b (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     449.1386 (1.0)      457.5788 (1.00)     453.7119 (1.0)      3.2048 (1.10)     453.2836 (1.0)      4.3319 (1.0)           2;0  2.2040 (1.0)           5           1
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         450.5393 (1.00)     457.4148 (1.0)      453.9127 (1.00)     2.9121 (1.0)      454.0299 (1.00)     5.1117 (1.18)          2;0  2.2031 (1.00)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

github-actions · 2026-02-07T19:48:40Z

Benchmark comparison for 140dccf7 (base) vs a3c9555e (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         443.6576 (1.0)      454.0091 (1.0)      447.3220 (1.0)      4.2598 (1.0)      446.6230 (1.0)      6.1454 (1.0)           1;0  2.2355 (1.0)           5           1
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     444.8959 (1.00)     461.4176 (1.02)     450.3497 (1.01)     6.6617 (1.56)     448.6970 (1.00)     8.1958 (1.33)          1;0  2.2205 (0.99)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

github-actions · 2026-02-07T22:16:13Z

Benchmark comparison for 140dccf7 (base) vs c9aa1cea (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         359.9217 (1.0)      366.7047 (1.0)      363.5997 (1.0)      2.8372 (1.0)      362.6679 (1.0)      4.5080 (1.0)           2;0  2.7503 (1.0)           5           1
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     364.2002 (1.01)     372.2619 (1.02)     368.2314 (1.01)     3.6488 (1.29)     367.3628 (1.01)     6.7145 (1.49)          2;0  2.7157 (0.99)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

codecov · 2026-02-07T22:22:20Z

Codecov Report

❌ Patch coverage is 83.78378% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
plugboard/tune/tune.py	84.61%	1 Missing and 3 partials ⚠️
plugboard/process/ray_process.py	75.00%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

github-actions · 2026-02-08T14:11:44Z

Benchmark comparison for 140dccf7 (base) vs a562b888 (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     451.9600 (1.0)      466.3536 (1.00)     460.1518 (1.0)      5.4607 (2.25)     461.5083 (1.00)     7.2211 (2.16)          2;0  2.1732 (1.0)           5           1
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         457.7223 (1.01)     464.1735 (1.0)      460.7200 (1.00)     2.4321 (1.0)      460.5332 (1.0)      3.3481 (1.0)           2;0  2.1705 (1.00)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

github-actions · 2026-02-11T19:54:02Z

Benchmark comparison for 140dccf7 (base) vs 82a76cff (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         441.6068 (1.0)      452.9054 (1.0)      447.1983 (1.0)      4.5385 (1.0)      445.5531 (1.0)      6.8884 (1.0)           2;0  2.2361 (1.0)           5           1
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     447.6334 (1.01)     461.5999 (1.02)     455.5029 (1.02)     5.4577 (1.20)     455.4675 (1.02)     8.0422 (1.17)          2;0  2.1954 (0.98)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

github-actions · 2026-02-11T19:57:03Z

Benchmark comparison for 140dccf7 (base) vs 2892fe9a (PR)


------------------------------------------------------------------------------------------------------------------- benchmark: 2 tests ------------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean             StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     443.1471 (1.0)      451.7052 (1.0)      447.0126 (1.0)       3.3056 (1.0)      446.2575 (1.0)       4.7749 (1.0)           2;0  2.2371 (1.0)           5           1
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         450.9760 (1.02)     474.8847 (1.05)     457.9494 (1.02)     10.0804 (3.05)     453.4598 (1.02)     12.3320 (2.58)          1;0  2.1836 (0.98)          5           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

chrisk314 · 2026-02-11T20:00:19Z

@copilot I have refactored the implementation of component resources so the resource requirements can now be declared with a class attribute Component.resources: Resource. On the base Component class a default value for the resources attribute is set using the default values from the Resource data class. Each instance of a component creates an instance scoped copy. This value is overridden with any resources value passed in to the Component constructor.

Update any relevant docstrings, documentation and examples to explain how to specify resource requirements on the Component declaration, and how this can be overriden.

Co-authored-by: chrisk314 <2366658+chrisk314@users.noreply.github.com>

Copilot · 2026-02-11T20:05:01Z

@copilot I have refactored the implementation of component resources so the resource requirements can now be declared with a class attribute Component.resources: Resource. On the base Component cl...

Done in commit ea310c1. Updated:

Component class docstring to document the resources class attribute
__init__ docstring to explain the resources parameter and how class-level defaults work
resources_example.py to demonstrate class-level resource declaration (recommended approach)
Documentation in running-in-parallel.md to explain both approaches:
- Class-level declaration (recommended for explicit component definitions)
- Constructor override (for flexibility when needed)
YAML example comments to clarify when class-level resources are being used vs overridden
Added comprehensive docstrings to all example component methods

chrisk314

lgtm

github-actions · 2026-02-12T10:42:44Z

Benchmark comparison for 140dccf7 (base) vs ea310c14 (PR)


------------------------------------------------------------------------------------------------------------------ benchmark: 2 tests -----------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                         Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_benchmark_process_run (pr/.benchmarks/Linux-CPython-3.12-64bit/0001_pr)         364.4447 (1.0)      368.0732 (1.0)      365.9193 (1.0)      1.5485 (1.03)     365.6114 (1.0)      2.6221 (1.11)          1;0  2.7328 (1.0)           5           1
test_benchmark_process_run (main/.benchmarks/Linux-CPython-3.12-64bit/0001_base)     367.1990 (1.01)     370.8392 (1.01)     368.9097 (1.01)     1.5080 (1.0)      369.4407 (1.01)     2.3657 (1.0)           2;0  2.7107 (0.99)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean

Initial plan

4530341

Copilot AI assigned Copilot and toby-coleman Jan 28, 2026

Copilot started work on behalf of toby-coleman January 28, 2026 19:44 View session

Copilot AI and others added 4 commits January 28, 2026 19:49

Add Resource class and integrate with Component, RayProcess, and Tuner

d529dee

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>

Add comprehensive tests for Resource implementation

f59376c

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>

Add examples for resource requirements in Ray processes

0219833

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>

Fix type annotations and address code review feedback

3ec7854

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add resource requirements for components~~ feat: Implement resource requirements for components Jan 28, 2026

Copilot finished work on behalf of toby-coleman January 28, 2026 20:01

Copilot AI requested a review from toby-coleman January 28, 2026 20:01

toby-coleman reviewed Jan 28, 2026

View reviewed changes

plugboard-schemas/plugboard_schemas/component.py Show resolved Hide resolved

toby-coleman reviewed Jan 28, 2026

View reviewed changes

plugboard-schemas/plugboard_schemas/component.py Outdated Show resolved Hide resolved

toby-coleman reviewed Jan 28, 2026

View reviewed changes

plugboard-schemas/plugboard_schemas/component.py Outdated Show resolved Hide resolved

toby-coleman reviewed Jan 28, 2026

View reviewed changes

plugboard/process/ray_process.py Outdated Show resolved Hide resolved

toby-coleman reviewed Jan 31, 2026

View reviewed changes

examples/tutorials/004_using_ray/resources-example.yaml Show resolved Hide resolved

toby-coleman reviewed Jan 31, 2026

View reviewed changes

plugboard/tune/tune.py Outdated Show resolved Hide resolved

toby-coleman requested changes Jan 31, 2026

View reviewed changes

Copilot started work on behalf of toby-coleman January 31, 2026 09:34 View session

Address code review feedback: generalize suffixes, make memory intege…

9515154

…r, move imports to top Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>

Copilot AI requested a review from toby-coleman January 31, 2026 09:38

Copilot finished work on behalf of toby-coleman January 31, 2026 09:38

toby-coleman added 8 commits February 3, 2026 21:26

Move suffixes to top

c400a97

Rework logic for tune placement bundles

a38aa88

Merge remote-tracking branch 'origin/main' into copilot/implement-res…

573a0e1

…ource-requirements

Docs config improved for schemas

31092ab

Tidy up documentation on resources example

6bf7017

Add resources for smoke tests

bae7c46

Make sure resources are available in runner

a9a4ef5

Remove non-existent cross refs in docs

c14cbaf

toby-coleman marked this pull request as ready for review February 7, 2026 18:33

toby-coleman approved these changes Feb 7, 2026

View reviewed changes

Typing

9b361df

Fixup test

d7c87c6

Increase tuner test timeout

a3c9555

Correct format for placement groups in Ray tune

c9aa1ce

toby-coleman added 2 commits February 8, 2026 14:07

Typo

155c571

Coverage

a562b88

chrisk314 added 4 commits February 11, 2026 20:36

refactor: Resource value validation

5fb86bc

fix: Inconsistent type annotation syntax

b0fd414

Allows declaring resources at Component declaration time

3f8031f

Enhances tests

82a76cf

Ignore lint error

2892fe9

Copilot started work on behalf of chrisk314 February 11, 2026 20:00 View session

Update docs and examples to show class-level resource declaration

ea310c1

Co-authored-by: chrisk314 <2366658+chrisk314@users.noreply.github.com>

Copilot AI requested a review from chrisk314 February 11, 2026 20:06

Copilot finished work on behalf of chrisk314 February 11, 2026 20:06

chrisk314 approved these changes Feb 11, 2026

View reviewed changes

feat: Implement resource requirements for components #194

Are you sure you want to change the base?

feat: Implement resource requirements for components #194

Uh oh!

Conversation

Copilot AI commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Example

Alternatives

Comments on the Issue (you are @copilot in this section)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

toby-coleman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 7, 2026

Uh oh!

github-actions bot commented Feb 7, 2026

Uh oh!

github-actions bot commented Feb 7, 2026

Uh oh!

github-actions bot commented Feb 7, 2026

Uh oh!

github-actions bot commented Feb 7, 2026

Uh oh!

codecov bot commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Feb 8, 2026

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

chrisk314 commented Feb 11, 2026

Uh oh!

Copilot AI commented Feb 11, 2026

Uh oh!

chrisk314 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jan 28, 2026 •

edited

Loading

codecov bot commented Feb 7, 2026 •

edited

Loading