\uac15\ud654\ud559\uc2b5(Reinforcement Learning, RL)\uc740 \uc5d0\uc774\uc804\ud2b8\uac00 \ud658\uacbd\uacfc \uc0c1\ud638\uc791\uc6a9\ud558\uba70 \ubcf4\uc0c1\uc744 \ucd5c\ub300\ud654\ud558\uae30 \uc704\ud574 \uc77c\ub828\uc758 \ud589\ub3d9\uc744 \ud559\uc2b5\ud558\ub294 \uae30\uacc4 \ud559\uc2b5\uc758 \ud55c \ubd84\uc57c\uc785\ub2c8\ub2e4. \uc774 \uacfc\uc815\uc5d0\uc11c \uc5d0\uc774\uc804\ud2b8\ub294 \uc0c1\ud0dc(state), \ud589\ub3d9(action), \ubcf4\uc0c1(reward)\uc758 \uc138 \uac00\uc9c0 \uc8fc\uc694 \uc694\uc18c\ub97c \uc0ac\uc6a9\ud558\uc5ec \ucd5c\uc801\uc758 \uc815\ucc45(policy)\uc744 \ud559\uc2b5\ud569\ub2c8\ub2e4. \ub525\ub7ec\ub2dd\uacfc \uacb0\ud569\ub41c \uac15\ud654\ud559\uc2b5, \uc989 \ub525 \uac15\ud654\ud559\uc2b5(Deep RL)\uc740 \uc7a0\uc7ac\uc801\uc73c\ub85c \ubcf5\uc7a1\ud55c \ud658\uacbd\uc5d0\uc11c\ub3c4 \ud6a8\uacfc\uc801\uc73c\ub85c \ud559\uc2b5\ud560 \uc218 \uc788\ub294 \ub2a5\ub825\uc744 \uc81c\uacf5\ud569\ub2c8\ub2e4.<\/p>\n
\ub525 Q-\ub124\ud2b8\uc6cc\ud06c(DQN)\ub294 \uac15\ud654\ud559\uc2b5\uc758 Q-\ub7ec\ub2dd(Q-learning) \uc54c\uace0\ub9ac\uc998\uc5d0 \ub525\ub7ec\ub2dd\uc744 \uc811\ubaa9\ud558\uc5ec \uace0\ucc28\uc6d0 \uc0c1\ud0dc \uacf5\uac04\uc5d0\uc11c \ud6a8\uacfc\uc801\uc73c\ub85c \uc791\ub3d9\ud560 \uc218 \uc788\ub3c4\ub85d \ud55c \uac83\uc785\ub2c8\ub2e4. Q-\ub7ec\ub2dd\uc740 \uc5b4\ub5a4 \uc0c1\ud0dc\uc5d0\uc11c \uc5b4\ub5a4 \ud589\ub3d9\uc744 \uc120\ud0dd\ud560 \ub54c\uc758 \uac00\uce58\ub97c \ucd94\uc815\ud558\uc5ec \ucd5c\uc801\uc758 \uc815\ucc45\uc744 \ud559\uc2b5\ud558\ub294 \ubc29\ubc95\uc785\ub2c8\ub2e4. DQN\uc740 \uc774 Q-\uac12\uc744 \uc608\uce21\ud558\uae30 \uc704\ud574 \uc2e0\uacbd\ub9dd\uc744 \uc0ac\uc6a9\ud569\ub2c8\ub2e4.<\/p>\n
Q-learning\uc740 \ub2e4\uc74c\uacfc \uac19\uc740 \uc5c5\ub370\uc774\ud2b8 \uaddc\uce59\uc744 \ub530\ub985\ub2c8\ub2e4:<\/p>\n
\nQ(s, a) <- Q(s, a) + \u03b1[r + \u03b3 max(Q(s', a')) - Q(s, a)]\n <\/code><\/pre>\n\uc5ec\uae30\uc11c:<\/p>\n
\ns<\/code>: \ud604\uc7ac \uc0c1\ud0dc<\/li>\na<\/code>: \ud604\uc7ac \ud589\ub3d9<\/li>\nr<\/code>: \ubcf4\uc0c1<\/li>\ns'<\/code>: \ub2e4\uc74c \uc0c1\ud0dc<\/li>\n\u03b1<\/code>: \ud559\uc2b5\ub960<\/li>\n\u03b3<\/code>: \ud560\uc778 \uacc4\uc218<\/li>\n<\/ul>\nQ\uac12\uc774 \uc2e4\uc81c\ub85c \uc5b4\ub5a4 \ud589\ub3d9\uc758 \uac00\uce58\ub97c \ub098\ud0c0\ub0b4\uac8c \ud558\uae30 \uc704\ud574 \uc624\ub79c \uc2dc\uac04\uc5d0 \uac78\uccd0 \ucda9\ubd84\ud55c \uacbd\ud5d8\uc744 \uc313\uc544\uc57c \ud558\uba70, \uc774 \uacfc\uc815\uc5d0\uc11c \ub9ce\uc740 \uba54\ubaa8\ub9ac\uc640 \uc2dc\uac04\uc774 \uc18c\ubaa8\ub429\ub2c8\ub2e4. \uc774 \ubb38\uc81c\ub97c \ud574\uacb0\ud558\uae30 \uc704\ud574 DQN\uc740 \uc2e0\uacbd\ub9dd\uc744 \uc0ac\uc6a9\ud558\uc5ec Q\uac12\uc744 \uadfc\uc0ac\ud569\ub2c8\ub2e4.<\/p>\n
2. DQN\uc758 \uad6c\uc870\uc640 \uad6c\uc131 \uc694\uc18c<\/h2>\n
DQN\uc758 \uc8fc\uc694 \uad6c\uc131 \uc694\uc18c\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4:<\/p>\n
\n- \uc2e0\uacbd\ub9dd(Neural Network):<\/strong> \uc0c1\ud0dc\ub97c \uc785\ub825\ubc1b\uc544 \uac01 \ud589\ub3d9\uc5d0 \ub300\ud55c Q\uac12\uc744 \ucd9c\ub825\ud569\ub2c8\ub2e4.<\/li>\n
- \uacbd\ud5d8 \uc7ac\ud50c\ub808\uc774(Experience Replay):<\/strong> \uc5d0\uc774\uc804\ud2b8\uac00 \uacfc\uac70 \uacbd\ud5d8\uc744 \ub2e4\uc2dc \ud65c\uc6a9\ud560 \uc218 \uc788\ub3c4\ub85d \ud558\ub294 \uae30\uc220\ub85c, \uc0d8\ud50c\uc744 \ubb34\uc791\uc704\ub85c \uc120\ud0dd\ud558\uc5ec Q\ud568\uc218\ub97c \ud559\uc2b5\ud569\ub2c8\ub2e4.<\/li>\n
- \ud0c0\uac9f \ub124\ud2b8\uc6cc\ud06c(Target Network):<\/strong> Q\uac12\uc758 \uc5c5\ub370\uc774\ud2b8\ub97c \uc548\uc815\ud654\ud558\uae30 \uc704\ud574 \uc8fc\uae30\uc801\uc73c\ub85c \uc5c5\ub370\uc774\ud2b8\ub418\ub294 \ubcc4\ub3c4\uc758 \uc2e0\uacbd\ub9dd\uc785\ub2c8\ub2e4.<\/li>\n<\/ul>\n
2.1 \uc2e0\uacbd\ub9dd\uc758 \uad6c\uc870<\/h3>\n
DQN\uc5d0\uc11c \uc0ac\uc6a9\ud558\ub294 \uc2e0\uacbd\ub9dd\uc740 \uc77c\ubc18\uc801\uc73c\ub85c \ub2e4\uc74c\uacfc \uac19\uc740 \ud615\ud0dc\ub97c \uac00\uc9d1\ub2c8\ub2e4:<\/p>\n
\nclass DQN(nn.Module):\n def __init__(self, input_size, output_size):\n super(DQN, self).__init__()\n self.fc1 = nn.Linear(input_size, 128)\n self.fc2 = nn.Linear(128, 128)\n self.fc3 = nn.Linear(128, output_size)\n\n def forward(self, x):\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n return self.fc3(x)\n <\/code><\/pre>\n\uc774 \uc2e0\uacbd\ub9dd\uc740 \uc785\ub825\uc73c\ub85c \uc8fc\uc5b4\uc9c4 \uc0c1\ud0dc\ub97c \ubc1b\uc544 Q\uac12\uc744 \uc608\uce21\ud558\ub294\ub370 \uc0ac\uc6a9\ub429\ub2c8\ub2e4.<\/p>\n
2.2 \uacbd\ud5d8 \uc7ac\ud50c\ub808\uc774(Experience Replay)<\/h3>\n
\uacbd\ud5d8 \uc7ac\ud50c\ub808\uc774\ub294 \uc5d0\uc774\uc804\ud2b8\uac00 \uc218\uc9d1\ud55c \uacbd\ud5d8\uc744 \uba54\ubaa8\ub9ac\uc5d0 \uc800\uc7a5\ud558\uace0, \uc774 \uba54\ubaa8\ub9ac\uc5d0\uc11c \ubb34\uc791\uc704\ub85c \uc0d8\ud50c\uc744 \uc120\ud0dd\ud558\uc5ec \ud559\uc2b5\ud569\ub2c8\ub2e4. \uc774\ub807\uac8c \ud558\uba74 \ub370\uc774\ud130\uc758 \uc0c1\uad00\uc131\uc744 \uc904\uc774\uace0, \ub354 \uc548\uc815\uc801\uc73c\ub85c \ud559\uc2b5\ud560 \uc218 \uc788\uac8c \ud574\uc90d\ub2c8\ub2e4.<\/p>\n
2.3 \ud0c0\uac9f \ub124\ud2b8\uc6cc\ud06c(Target Network)<\/h3>\n
\ud0c0\uac9f \ub124\ud2b8\uc6cc\ud06c\ub294 \uc8fc \ub124\ud2b8\uc6cc\ud06c\uc640 \ub3d9\uc77c\ud55c \uad6c\uc870\ub97c \uac00\uc9c4 \uc2e0\uacbd\ub9dd\uc73c\ub85c, \uc77c\uc815 \uc8fc\uae30\ub9c8\ub2e4 \uc8fc \ub124\ud2b8\uc6cc\ud06c\uc758 \uac00\uc911\uce58\ub97c \ubcf5\uc0ac\ud558\uc5ec \uc5c5\ub370\uc774\ud2b8\ub429\ub2c8\ub2e4. \uc774\ub97c \ud1b5\ud574 Q\uac12\uc758 \uc5c5\ub370\uc774\ud2b8\ub97c \ub354 \uc548\uc815\uc801\uc73c\ub85c \ub9cc\ub4e4\uc5b4 \uc90d\ub2c8\ub2e4.<\/p>\n
3. DQN\uc758 \ud559\uc2b5 \uacfc\uc815<\/h2>\n
DQN\uc758 \ud559\uc2b5 \uacfc\uc815\uc740 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4:<\/p>\n
\n- \ud658\uacbd\uc73c\ub85c \ubd80\ud130 \ucd08\uae30 \uc0c1\ud0dc
s<\/code>\ub97c \ubc1b\uc544\uc635\ub2c8\ub2e4.<\/li>\n- \ubc30\uce58 \ud06c\uae30\ub9cc\ud07c \ubb34\uc791\uc704 \ud589\ub3d9\uc744 \uc120\ud0dd\ud558\uace0 \uacbd\ud5d8\uc744 \uba54\ubaa8\ub9ac\uc5d0 \uc800\uc7a5\ud569\ub2c8\ub2e4.<\/li>\n
- \uc5d0\uc774\uc804\ud2b8\uac00 \uc120\ud0dd\ud55c \ud589\ub3d9
a<\/code>\ub97c \ud658\uacbd\uc5d0 \uc801\uc6a9\ud558\uc5ec \uc0c8\ub85c\uc6b4 \uc0c1\ud0dc s'<\/code>\uc640 \ubcf4\uc0c1 r<\/code>\uc744 \uad00\ucc30\ud569\ub2c8\ub2e4.<\/li>\n- \uacbd\ud5d8 \uc7ac\ud50c\ub808\uc774\ub97c \ud1b5\ud574 \uc0d8\ud50c\uc744 \ubb34\uc791\uc704\ub85c \uc120\ud0dd\ud558\uace0, Q\ud568\uc218\ub97c \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.<\/li>\n
- \uc0c8\ub85c\uc6b4 \uc0c1\ud0dc
s'<\/code>\uc5d0\uc11c \ub2e4\uc2dc \ud589\ub3d9\uc744 \uc120\ud0dd\ud558\uace0, \uc774\ub97c \ubc18\ubcf5\ud569\ub2c8\ub2e4.<\/li>\n<\/ol>\n4. DQN\uc758 \uc608\uc81c \ucf54\ub4dc<\/h2>\n
\uc774\uc81c DQN\uc758 \uad6c\ud604 \uc608\uc81c\ub97c \uc0b4\ud3b4\ubd05\uc2dc\ub2e4. \ub2e4\uc74c \ucf54\ub4dc\ub294 Python\uc744 \uc0ac\uc6a9\ud558\uc5ec OpenAI\uc758 Gym \ud658\uacbd\uc5d0\uc11c DQN\uc744 \uad6c\ud604\ud558\ub294 \uc608\uc81c\uc785\ub2c8\ub2e4.<\/p>\n
\nimport random\nimport numpy as np\nimport gym\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\n\n# DQN \ubaa8\ub378 \uc815\uc758\nclass DQN(nn.Module):\n def __init__(self, input_size, output_size):\n super(DQN, self).__init__()\n self.fc1 = nn.Linear(input_size, 128)\n self.fc2 = nn.Linear(128, 128)\n self.fc3 = nn.Linear(128, output_size)\n\n def forward(self, x):\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n return self.fc3(x)\n\n# \uacbd\ud5d8 \uc7ac\ud50c\ub808\uc774\ub97c \uc704\ud55c \ub9ac\ud50c\ub808\uc774 \uba54\ubaa8\ub9ac\nclass ReplayMemory:\n def __init__(self, capacity):\n self.memory = []\n self.capacity = capacity\n \n def push(self, transition):\n if len(self.memory) < self.capacity:\n self.memory.append(transition)\n else:\n self.memory.pop(0)\n self.memory.append(transition)\n \n def sample(self, batch_size):\n return random.sample(self.memory, batch_size)\n \n def __len__(self):\n return len(self.memory)\n\n# \ud558\uc774\ud37c\ud30c\ub77c\ubbf8\ud130\nBATCH_SIZE = 32\nGAMMA = 0.99\nEPSILON_START = 1.0\nEPSILON_END = 0.1\nEPSILON_DECAY = 200\nTARGET_UPDATE = 10\nMEMORY_CAPACITY = 10000\nNUM_EPISODES = 1000\n\n# \ud658\uacbd \ucd08\uae30\ud654\nenv = gym.make('CartPole-v1')\nn_actions = env.action_space.n\nn_obs = env.observation_space.shape[0]\n\n# DQN \ubaa8\ub378 \uc0dd\uc131\npolicy_net = DQN(n_obs, n_actions)\ntarget_net = DQN(n_obs, n_actions)\ntarget_net.load_state_dict(policy_net.state_dict())\ntarget_net.eval()\n\noptimizer = optim.Adam(policy_net.parameters())\nmemory = ReplayMemory(MEMORY_CAPACITY)\n\n# epsilon-greedy \ud589\ub3d9 \uc120\ud0dd\ndef select_action(state):\n global steps_done\n eps_threshold = EPSILON_END + (EPSILON_START - EPSILON_END) * \\\n np.exp(-1. * steps_done \/ EPSILON_DECAY)\n steps_done += 1\n if random.random() > eps_threshold:\n with torch.no_grad():\n return policy_net(torch.FloatTensor(state)).max(0)[1].item() # \ucd5c\uc801 \ud589\ub3d9 \uc120\ud0dd\n else:\n return random.randrange(n_actions) # \ub79c\ub364 \ud589\ub3d9 \uc120\ud0dd\n\n# \ud559\uc2b5 \uc8fc\uae30\ndef optimize_model():\n if len(memory) < BATCH_SIZE:\n return\n transitions = memory.sample(BATCH_SIZE)\n batch = list(zip(*transitions)) # \ubc30\uce58 \ubd84\ub9ac\n state_batch = torch.FloatTensor(batch[0])\n action_batch = torch.LongTensor(batch[1]).unsqueeze(1)\n reward_batch = torch.FloatTensor(batch[2])\n next_state_batch = torch.FloatTensor(batch[3])\n \n # Q\uac12 \uacc4\uc0b0\n state_action_values = policy_net(state_batch).gather(1, action_batch)\n next_state_values = target_net(next_state_batch).max(1)[0].detach()\n expected_state_action_values = reward_batch + (GAMMA * next_state_values)\n \n # \uc190\uc2e4 \uacc4\uc0b0\n loss = nn.functional.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n# \uba54\uc778 \ud559\uc2b5 \ub8e8\ud504\nsteps_done = 0\nfor episode in range(NUM_EPISODES):\n state = env.reset()\n for t in range(1000):\n action = select_action(state)\n next_state, reward, done, _ = env.step(action)\n memory.push((state, action, reward, next_state))\n state = next_state\n optimize_model()\n if done:\n break\n if episode % TARGET_UPDATE == 0:\n target_net.load_state_dict(policy_net.state_dict())\n print(f\"Episode {episode + 1}\/{NUM_EPISODES} completed.\")\n <\/code><\/pre>\n5. DQN\uc758 \ud55c\uacc4\uc640 \uac1c\uc120 \ubc29\uc548<\/h2>\n
DQN\uc740 \uba87 \uac00\uc9c0 \ud55c\uacc4\ub97c \uac00\uc9c0\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc6b0\uc120, \uc2a4\ud06c\ub9ac\ub529 \ubb38\uc81c(specific problem)\uc5d0 \ub300\ud574 \uc798 \uc218\ub834\ud558\uc9c0 \ubabb\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \ubb38\uc81c\ub294 \ud2b9\uc815 \uc0c1\ud0dc\ub098 \ud589\ub3d9\uc5d0 \ub300\ud574\uc11c\ub9cc \uc9c0\ub098\uce58\uac8c \ub192\uc740 Q\uac12\uc744 \uc608\uce21\ud558\ub294 \ud604\uc0c1\uc785\ub2c8\ub2e4. \uc774\ub97c \ud574\uacb0\ud558\uae30 \uc704\ud574 \uc5ec\ub7ec \uac00\uc9c0 \uac1c\uc120 \ubc29\ubc95\uc774 \uc5f0\uad6c\ub418\uc5c8\uc2b5\ub2c8\ub2e4:<\/p>\n
\n- Double DQN:<\/strong> Q-\ub7ec\ub2dd\uc758 \uac15\ud55c \ube44\ud3b8\ud5a5(propensity)\uc744 \ud574\uacb0\ud558\uae30 \uc704\ud574, \ub450 \uac1c\uc758 Q-\ub124\ud2b8\uc6cc\ud06c\ub97c \uc0ac\uc6a9\ud558\uc5ec \uac01\uac01 Q\uac12\uc744 \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.<\/li>\n
- Dueling DQN:<\/strong> \uc0c1\ud0dc\uc640 \ud589\ub3d9\uc758 \uac00\uce58\ub97c \ubd84\ub9ac\ud558\uc5ec \uc124\uacc4\ub41c \uad6c\uc870\ub85c, \ubaa8\ub378\uc774 \ub354 \uc88b\uc740 \uc131\ub2a5\uc744 \ubc1c\ud718\ud558\ub3c4\ub85d \ud569\ub2c8\ub2e4.<\/li>\n
- Prioritized Experience Replay:<\/strong> \uc911\uc694\ub3c4\uac00 \ub192\uc740 \uacbd\ud5d8\uc744 \ub354 \uc790\uc8fc \uc0d8\ud50c\ub9c1\ud558\uc5ec \ud559\uc2b5 \ud6a8\uc728\uc744 \ub192\uc785\ub2c8\ub2e4.<\/li>\n<\/ul>\n
6. \uacb0\ub860<\/h2>\n
\ub525 Q-\ub124\ud2b8\uc6cc\ud06c(DQN)\ub294 \ubcf5\uc7a1\ud55c \ud658\uacbd\uc5d0\uc11c \ud6a8\uacfc\uc801\uc73c\ub85c \uac15\ud654\ud559\uc2b5 \ubb38\uc81c\ub97c \ud574\uacb0\ud560 \uc218 \uc788\ub294 \uac15\ub825\ud55c \ub3c4\uad6c\uc785\ub2c8\ub2e4. \uc774 \uac15\uc88c\uc5d0\uc11c \uc18c\uac1c\ub41c \uae30\ucd08 \uac1c\ub150\uacfc \uc608\uc81c \ucf54\ub4dc\ub97c \ud1b5\ud574 DQN\uc758 \uc791\ub3d9 \ubc29\uc2dd\uc744 \uc774\ud574\ud558\uace0, \uc9c1\uc811 \uad6c\ud604\ud574\ubcfc \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uac15\uc88c\ub97c \ud1b5\ud574 \ub525 \uac15\ud654\ud559\uc2b5\uc758 \uae30\ucd08\ub97c \uc775\ud600 \ub2e4\uc74c \ub2e8\uacc4\ub85c \ub098\uc544\uac08 \uc218 \uc788\uae30\ub97c \ubc14\ub78d\ub2c8\ub2e4.<\/p>\n
\ucc38\uace0\uc790\ub8cc<\/h3>\n