Prioritized Buffer

Ver. 1.0.1 (2021-09-26)

This module provides the Prioritized Buffer based on the reference.

class mlpro.rl.pool.sarsbuffer.PrioritizedBuffer.PrioritizedBufferElement(p_state: State, p_action: Action, p_reward: Reward, p_state_new: State)

Bases: SARSElement

Element of a State-Action-Reward-Buffer.

class mlpro.rl.pool.sarsbuffer.PrioritizedBuffer.PrioritizedBuffer(p_size=1, alpha: float = 0.3, beta: float = 1)

Bases: SARSBuffer

Prioritized Sampling State-Action-Reward-Buffer in dictionary.

add_element(p_elem: PrioritizedBufferElement)

Add element to the buffer.

Parameters:

p_elem (BufferElement) – Element of Buffer

_gen_sample_ind(p_num: int) → list

Generate random indices from the buffer.

Parameters:

p_num (int) – Number of sample

Returns:

List of incides

_extract_rows(p_list_idx: list)

Extract the element in the buffer based on a list of indices.

Parameters:

p_list_idx (list) – List of indices

Returns:

Samples in dictionary

get_latest()

Returns latest buffered element.

get_all()

Return all buffered elements.

update_priorities(p_list_idx: list, priorities: ndarray)

Updates the priority tree. Needs to be called during each training step, utilising the element-wise calculated loss.

class mlpro.rl.pool.sarsbuffer.PrioritizedBuffer.SegmentTree(capacity: int, operation: Callable, init_value: float)

Bases: object

Reference: https://github.com/openai/baselines/blob/master/baselines/common/segment_tree.py .. attribute:: capacity

type:

int

tree

Type:

list

operation

Type:

function

_operate_helper(start: int, end: int, node: int, node_start: int, node_end: int) → float

Returns result of operation in segment.

operate(start: int = 0, end: int = 0) → float

Returns result of applying ‘self.operation’.

class mlpro.rl.pool.sarsbuffer.PrioritizedBuffer.SumSegmentTree(capacity: int)

Bases: SegmentTree

Reference: https://github.com/openai/baselines/blob/master/baselines/common/segment_tree.py

sum(start: int = 0, end: int = 0) → float

Returns arr[start] + … + arr[end].

retrieve(upperbound: float) → int

Find the highest index i about upper bound in the tree

class mlpro.rl.pool.sarsbuffer.PrioritizedBuffer.MinSegmentTree(capacity: int)

Bases: SegmentTree

Reference: https://github.com/openai/baselines/blob/master/baselines/common/segment_tree.py

min(start: int = 0, end: int = 0) → float

Returns min(arr[start], …, arr[end]).