Batch size once again

added Bug label

assigned to @dl337788

changed milestone to %Official dates

My implementation of batch handling might deviate from the OpenAI implementation. Mainly the number of updates performed and the size of mini batches might differ.

marked this issue as related to #72 (closed)

added In Progress label

Property	OpenAI	Me	Kostrikov
time steps	10_000_000	10_000_000	10_000_000
rollout length	128	128	128
number of envs	8	8	8
total rollout items (batch size)	128 * 8 = 1024	same	same
number of epochs	4	3	4
number of mini batches	4	3	32
mini batches per epoch	4	1	32
number of gradient updates	4 * 4 = 16	3	number of epochs * number of mini batches = 128
data used per mini batch	25%	25%	32
mini batch size	batch size / number of mini batches = 256	rollout length * data used per batch * num_envs = 32 * 8 = 256	32
number of updates	time steps / total rollout items = 9765	time steps / rollout length = 78_125	time steps / rollout length / number of envs = 9765

Both OpenAI and Kostrikov use the full dataset in each PPO epoch.

closed

reopened

mentioned in commit d48816c6

mentioned in commit 0c458e36

added In Review label and removed In Progress label

closed

removed In Review label