\uc6b0\uc120\uc21c\uc704\ub97c \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.<\/li>\n<\/ol>\n3.4 PER\uc758 \uc7a5\uc810\uacfc \ub2e8\uc810<\/h3>\n
PER\uc740 \uc5d0\uc774\uc804\ud2b8\uac00 \uc911\uc694\ud55c \uacbd\ud5d8\uc744 \ub354 \ub9ce\uc774 \ud559\uc2b5\ud560 \uc218 \uc788\ub3c4\ub85d \ub3d5\uae30 \ub54c\ubb38\uc5d0, \ub354 \ub098\uc740 \ud559\uc2b5 \uc18d\ub3c4\ub97c \uc790\ub791\ud569\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \ud544\uc5f0\uc801\uc73c\ub85c \ucd94\uac00\uc801\uc778 \uacc4\uc0b0 \ube44\uc6a9\uc774 \ubc1c\uc0dd\ud558\uba70, \uc0c8\ub86d\uac8c \uc0d8\ud50c\ub9c1\ub41c \uc6b0\uc120\uc21c\uc704\ub97c \uc5c5\ub370\uc774\ud2b8\ud574\uc57c \ud558\ub294 \ubcf5\uc7a1\uc131\uc774 \uac00\uc911\ub429\ub2c8\ub2e4.<\/p>\n
4. \ucd94\uac00\uc801\uc778 \uace0\uae09 \uae30\ubc95<\/h2>\n
\uc774\uc81c \uacbd\ud5d8 \uc7ac\uc0dd \ubc0f PER \uc678\uc5d0 \ub2e4\ub978 \uc0d8\ud50c \ud6a8\uc728\uc131\uc744 \ub192\uc774\uae30 \uc704\ud574 \uc0ac\uc6a9\ub418\ub294 \uc5ec\ub7ec \uae30\ubc95\ub4e4\uc744 \uc0b4\ud3b4\ubcf4\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n
4.1 Double Q-Learning<\/h3>\n
Double Q-Learning\uc740 Q-\uac12\uc758 \uc624\ubc84\uc5d0\uc2a4\ud2f0\uba54\uc774\uc158 \ubb38\uc81c\ub97c \ud574\uacb0\ud558\uae30 \uc704\ud574 \uc81c\uc548\ub41c \uae30\uc220\uc785\ub2c8\ub2e4. \uc77c\ubc18\uc801\uc778 Q-learning\uc5d0\uc11c\ub294 \ucd5c\uc801\uc758 \ud589\ub3d9\uc744 \uc120\ud0dd\ud558\uae30 \uc704\ud574 Q-\ud14c\uc774\ube14\uc744 \uc5c5\ub370\uc774\ud2b8 \ud569\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \uc774 \uacfc\uc815\uc5d0\uc11c Q-\uac12\uc774 \uacfc\ub300\ud3c9\uac00\ub418\ub294 \uacbd\ud5a5\uc774 \uc788\uc2b5\ub2c8\ub2e4. Double Q-Learning\uc740 \ub450 \uac1c\uc758 Q-\ud14c\uc774\ube14\uc744 \uc0ac\uc6a9\ud558\uc5ec, \ud604\uc7ac \ud589\ub3d9\uc744 \uc120\ud0dd\ud558\uae30 \uc704\ud574 \ud55c \ud14c\uc774\ube14\uc744 \uc559\uc0c1\ube14\ud558\uace0, \ub2e4\ub978 \ud14c\uc774\ube14\uc744 \uc0ac\uc6a9\ud558\uc5ec \uc5c5\ub370\uc774\ud2b8\ud558\uc5ec \uacfc\ub300\ud3c9\uac00 \ubb38\uc81c\ub97c \uc904\uc785\ub2c8\ub2e4.<\/p>\n
4.2 Dueling Network Architectures<\/h3>\n
Dueling Network\ub294 Q-value\ub97c \uc0c1\ud0dc \uac00\uce58\uc640 Advantage\ub85c \ubd84\ub9ac\ud55c \uad6c\uc870\uc785\ub2c8\ub2e4. \uc774\ub97c \ud1b5\ud574 \uc5d0\uc774\uc804\ud2b8\ub294 \uc0c1\ud0dc\uc758 \uac00\uce58\uc640 \ud2b9\uc815 \ud589\ub3d9\uc758 \uc0c1\ub300\uc801\uc778 \uc774\uc810\uc744 \ub3d9\uc2dc\uc5d0 \ud559\uc2b5\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \uad6c\uc870\ub294 \ud2b9\ud788 \uc120\ud0dd\ud560 \ud589\ub3d9\uc758 \uc218\uac00 \ub9ce\uc740 \ubcf5\uc7a1\ud55c \ud658\uacbd\uc5d0\uc11c \uc0d8\ud50c \ud6a8\uc728\uc131\uc744 \ud5a5\uc0c1\uc2dc\ud0a4\ub294 \ub370 \ub3c4\uc6c0\uc774 \ub429\ub2c8\ub2e4.<\/p>\n
4.3 \uc0ac\uc6a9 \uac00\ub2a5\ud55c \uc778\uc9c0\uc801 \uae30\ubc95<\/h3>\n
\uacfc\uac70 \uacbd\ud5d8\uc744 \ubb34\uc791\uc704\ub85c \uc120\ud0dd\ud558\ub294 \ub300\uc2e0, \uc778\uc9c0\uc801 \uae30\ubc95\uc744 \uc0ac\uc6a9\ud558\uc5ec \uac01 \uc0d8\ud50c\uc758 \uc911\uc694\uc131\uc744 \ud3c9\uac00\ud558\uace0, \uadf8\uc5d0 \ub530\ub77c \uc120\ud0dd\uc801\uc778 \ud559\uc2b5\uc744 \uc218\ud589\ud558\ub294 \ubc29\ubc95\ub3c4 \uc874\uc7ac\ud569\ub2c8\ub2e4. \uc774 \uc811\uadfc\uc740 \uc0d8\ud50c\uc758 \uc7ac\uc0ac\uc6a9 \ube48\ub3c4\ub97c \ub192\uc774\uba70, \ud559\uc2b5 \uacfc\uc815\uc5d0\uc11c\uc758 \uc0d8\ud50c \ubd88\uade0\ud615 \ubb38\uc81c\ub97c \uc904\uc774\ub294 \ub370 \uae30\uc5ec\ud569\ub2c8\ub2e4.<\/p>\n
5. \uc0ac\ub840 \uc5f0\uad6c: OpenAI Gym\uc5d0\uc11c\uc758 PER \uad6c\ud604<\/h2>\n
\uc774\uc81c OpenAI Gym \ud658\uacbd\uc5d0\uc11c Prioritized Experience Replay\ub97c \uc2e4\uc81c\ub85c \uad6c\ud604\ud574\ubcf4\ub294 \uc608\uc81c\ub97c \uc124\uba85\ud558\uaca0\uc2b5\ub2c8\ub2e4. \uc774 \uc608\uc81c\uc5d0\uc11c\ub294 CartPole \ud658\uacbd\uc744 \uc0ac\uc6a9\ud558\uc5ec \uac04\ub2e8\ud55c DQN \uc5d0\uc774\uc804\ud2b8\ub97c \uc791\uc131\ud560 \uac83\uc785\ub2c8\ub2e4.<\/p>\n
\n \n import numpy as np\n import random\n import gym\n from collections import deque\n \n class PrioritizedReplayBuffer:\n def __init__(self, capacity, alpha=0.6):\n self.capacity = capacity\n self.alpha = alpha\n self.buffer = []\n self.priorities = []\n self.position = 0\n \n def push(self, transition):\n max_priority = max(self.priorities, default=1.0) if self.priorities else 1.0\n if len(self.buffer) < self.capacity:\n self.buffer.append(transition)\n self.priorities.append(max_priority)\n else:\n self.buffer[self.position] = transition\n self.priorities[self.position] = max_priority\n \n self.position = (self.position + 1) % self.capacity\n \n def sample(self, batch_size, beta=0.4):\n priorities = np.array(self.priorities)\n probabilities = priorities ** self.alpha\n probabilities \/= probabilities.sum()\n indices = np.random.choice(len(self.buffer), size=batch_size, p=probabilities)\n \n samples = [self.buffer[idx] for idx in indices]\n total_samples = len(self.buffer)\n weights = (total_samples * probabilities[indices]) ** -beta\n weights \/= weights.max()\n \n return samples, indices, weights\n \n def update_priorities(self, indices, priorities):\n for idx, priority in zip(indices, priorities):\n self.priorities[idx] = priority\n \n def main():\n env = gym.make('CartPole-v1')\n replay_buffer = PrioritizedReplayBuffer(10000)\n num_episodes = 1000\n \n for episode in range(num_episodes):\n state = env.reset()\n done = False\n \n while not done:\n # \ud589\ub3d9 \uc120\ud0dd \ub85c\uc9c1\n action = env.action_space.sample()\n next_state, reward, done, _ = env.step(action)\n \n # \uacbd\ud5d8 \ucd94\uac00\n replay_buffer.push((state, action, reward, next_state, done))\n \n # \uc0d8\ud50c\ub9c1\n if len(replay_buffer.buffer) >= 32:\n batch, indices, weights = replay_buffer.sample(32)\n # \ud559\uc2b5 \ub85c\uc9c1\n # ...\n \n # \uc6b0\uc120\uc21c\uae30 \uc5c5\ub370\uc774\ud2b8\n # ...\n \n state = next_state\n \n print(f'Episode {episode} finished')\n \n if __name__ == \"__main__\":\n main()\n <\/code>\n <\/pre>\n\uacb0\ub860<\/h2>\n
\ubcf8 \ud3ec\uc2a4\ud2b8\uc5d0\uc11c\ub294 \uacbd\ud5d8 \uc7ac\uc0dd\uacfc \uc0d8\ud50c \ud6a8\uc728\uc131\uc744 \ud5a5\uc0c1\uc2dc\ud0a4\uae30 \uc704\ud55c \ub2e4\uc591\ud55c \uae30\ubc95\ub4e4, \ud2b9\ud788 Prioritized Experience Replay\uc5d0 \ub300\ud574 \uc54c\uc544\ubcf4\uc558\uc2b5\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uae30\ubc95\ub4e4\uc740 \uac15\ud654 \ud559\uc2b5\uc758 \ud6a8\uc728\uc131\uc744 \ub192\uc5ec\uc904 \ubfd0\ub9cc \uc544\ub2c8\ub77c, \ub354\uc6b1 \ubcf5\uc7a1\ud55c \ud658\uacbd\uc5d0\uc11c\ub3c4 \uc548\uc815\uc131\uacfc \uc131\ub2a5\uc744 \ud5a5\uc0c1\uc2dc\ud0ac \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ud5a5\ud6c4 \uc9c0\uc18d\uc801\uc778 \uc5f0\uad6c\uc640 \uc2e4\ud5d8\uc744 \ud1b5\ud574 \uc774\ub7ec\ud55c \uae30\ubc95\ub4e4\uc744 \ub354 \ubc1c\uc804\uc2dc\ucf1c \ud6a8\uacfc\uc801\uc778 \uac15\ud654 \ud559\uc2b5 \uc194\ub8e8\uc158\uc744 \ub9cc\ub4e4\uc5b4 \ub098\uac00\uae38 \uae30\ub300\ud569\ub2c8\ub2e4.<\/p>\n