Does it support reinforcement learning training with a single GPU and sglang?

[xinyubai1209]

I have an NVIDIA A800 (80G) GPU and would like to use areal to train a tiny llm. Does areal support this scenario, and what are the key configurations required? Thank you very much!

[xinyubai1209]

It seems that this configuration cannot be used properly. After sglang loads the model parameters, the areal framework remains unresponsive for a long time...

experiment_name: gsm8k-grpo
trial_name: trial0
allocation_mode: sglang.d1p1t1
cluster:
  fileroot: /tmp/areal/experiments
  n_nodes: 1
  n_gpus_per_node: 1
  name_resolve:
    type: nfs
    nfs_record_root: /tmp/areal/name_resolve
seed: 1
total_train_epochs: 5
tokenizer_path: ${actor.path}
async_training: true

rollout:
  experiment_name: ${experiment_name}
  trial_name: ${trial_name}
  max_concurrent_rollouts: 128
  queue_size: null
  consumer_batch_size: ${train_dataset.batch_size}
  max_head_offpolicyness: 2
  enable_rollout_tracing: false

gconfig:
  n_samples: 4
  min_new_tokens: 0
  max_new_tokens: 1024
  greedy: false
  temperature: 1.0

actor:
  experiment_name: ${experiment_name}
  trial_name: ${trial_name}
  path: /home/Qwen2.5-1.5B-Instruct
  init_from_scratch: false
  disable_dropout: true
  gradient_checkpointing: false
  dtype: bfloat16
  mb_spec:
    max_tokens_per_mb: 10240
  optimizer:
    type: adam
    lr: 1.70e-5
    weight_decay: 0.017
    beta1: 0.9
    beta2: 0.999
    eps: 1e-8
    lr_scheduler_type: constant
    gradient_clipping: 1.0
    warmup_steps_proportion: 0.001
  backend: fsdp
  sglang:
    hybrid_train: true

  group_size: ${gconfig.n_samples}
  group_adv_norm: false
  eps_clip: 0.4
  temperature: ${gconfig.temperature}
  reward_scaling: 10.0
  reward_bias: -0.5
  kl_ctl: 0.0
  ppo_n_minibatches: 1
  recompute_logprob: true
  use_decoupled_loss: true
  behav_imp_weight_cap: 5.0
  dynamic_sampling: false

ref:
  experiment_name: ${experiment_name}
  trial_name: ${trial_name}
  path: ${actor.path}
  init_from_scratch: false
  disable_dropout: true
  dtype: ${actor.dtype}
  mb_spec:
    max_tokens_per_mb: 10240
  optimizer: null
  backend: fsdp

# SGLang
sglang:
  model_path: ${actor.path}
  random_seed: ${seed}
  skip_tokenizer_init: true
  dtype: ${actor.dtype}
  max_running_requests: null
  context_length: 32768
  mem_fraction_static: 0.4

# datasets
train_dataset:
  batch_size: 64
  shuffle: true
  pin_memory: true
  num_workers: 4
  path: openai/gsm8k
  type: rl
  max_length: 1024

valid_dataset:
  batch_size: 64
  shuffle: true
  pin_memory: true
  num_workers: 4
  path: openai/gsm8k
  type: rl

# Utilities
saver:
  experiment_name: ${experiment_name}
  trial_name: ${trial_name}
  fileroot: ${cluster.fileroot}
  freq_epochs: 1
  freq_steps: null
  freq_secs: null

recover:
  mode: disabled
  experiment_name: ${experiment_name}
  trial_name: ${trial_name}
  fileroot: ${cluster.fileroot}
  freq_epochs: 1
  freq_steps: null
  freq_secs: 3600

evaluator:
  experiment_name: ${experiment_name}
  trial_name: ${trial_name}
  fileroot: ${cluster.fileroot}
  freq_epochs: 1
  freq_steps: null
  freq_secs: null

stats_logger:
  experiment_name: ${experiment_name}
  trial_name: ${trial_name}
  fileroot: ${cluster.fileroot}
  wandb:
    mode: disabled

launcher:
  inference_server_cpus_per_gpu: 4
  inference_server_mem_per_gpu: 32768
  trainer_cpus_per_gpu: 4
  trainer_mem_per_gpu: 32768

[garrett4wade]

Hi @xinyubai1209 , currently NO.

sglang.d1 is the allocation mode to start sglang servers only. The colocate allocation should be something like sglang:d1 | fsdp:d1, but we have not implemented it for now.

The colocation mode is on our 2025 Q3 roadmap. Please stay tuned.

[xinyubai1209]

Hi @xinyubai1209 , currently NO.

sglang.d1 is the allocation mode to start sglang servers only. The colocate allocation should be something like sglang:d1 | fsdp:d1, but we have not implemented it for now.

The colocation mode is on our 2025 Q3 roadmap. Please stay tuned.

Thanks for your answer, looking forward to it!