Replay Buffer

Replay buffer Class

class modularl.replay_buffers.ReplayBuffer(buffer_size: int, sampling='random', **kwargs)

Bases: AbstractReplayBuffer

A replay buffer for storing and sampling transitions for reinforcement learning.

This class is a wrapper around the TorchRL replay buffers. It provides a simple interface for storing and sampling transitions.

Parameters:

• buffer_size (int) – The maximum capacity of the replay buffer.

• sampling (str) – The type of sampling to use. Options are ‘random’, ‘prioritized’, and ‘without_replacement’.

• **kwargs – Additional keyword arguments to be passed to the base class.

Attributes:

storage (LazyMemmapStorage): The storage object for storing transitions. sampler (Sampler): The sampler object for sampling transitions. buffer (TensorDictReplayBuffer): The buffer object for managing the storage and sampling.

Note

This class is a wrapper around the TorchRL replay buffers. For more advanced usage and configurations, you can use the TorchRL replay buffers directly. Refer to the TorchRL replay buffer tutorial for more details.

sample(batch_size: int)

Sample a batch of transitions from the buffer.

Parameters:

batch_size (int) – The number of transitions to sample.

Returns:

A batch of sampled transitions.

Return type:

TensorDict

extend(transition)

Add a transition to the buffer.

Parameters:

transition (TensorDict) – The transition to add to the buffer.

update(idx, transition)

Update a transition in the buffer.

Parameters:

• idx (int) – The index of the transition to update.

• transition (TensorDict) – The updated transition.

Example Usage

Here’s an example of how to use the replay buffer: