\uac15\ud654\ud559\uc2b5(reinforcement learning, RL)\uc740 \uae30\uacc4 \ud559\uc2b5\uc758 \ud55c \ubd84\uc57c\ub85c, \uc5d0\uc774\uc804\ud2b8\uac00 \ud658\uacbd\uacfc \uc0c1\ud638\uc791\uc6a9\ud558\uba70 \ucd5c\uc801\uc758 \uc815\ucc45\uc744 \ud559\uc2b5\ud558\ub294 \ubc29\ubc95\uc785\ub2c8\ub2e4. \uc774 \uae00\uc5d0\uc11c\ub294 \uc815\ucc45 \uae30\ubc18 \uac15\ud654\ud559\uc2b5\uc758 \ud55c \ud615\ud0dc\uc778 Proximal Policy Optimization (PPO)<\/strong>, Asynchronous Actor-Critic (A3C)<\/strong>, \uadf8\ub9ac\uace0 Soft Actor-Critic (SAC)<\/strong> \uc54c\uace0\ub9ac\uc998\uc5d0 \ub300\ud574 \uae4a\uc774 \uc788\uac8c \uc0b4\ud3b4\ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \uac01 \uc54c\uace0\ub9ac\uc998\uc758 \ud2b9\uc9d5\uacfc \uc2e4\uc81c \uc801\uc6a9 \uc0ac\ub840\ub97c \ubd84\uc11d\ud558\uc5ec, \uc774\ub860\uacfc \uc2e4\uc2b5\uc758 \ub2e4\ub9ac\ub97c \ub193\uace0\uc790 \ud569\ub2c8\ub2e4.<\/p>\n \uc815\ucc45 \uae30\ubc18 \uac15\ud654\ud559\uc2b5\uc5d0\uc11c\ub294 \uc5d0\uc774\uc804\ud2b8\uac00 \ud2b9\uc815 \uc0c1\ud669\uc5d0\uc11c \uc5b4\ub5a4 \ud589\ub3d9\uc744 \uc120\ud0dd\ud560\uc9c0\ub97c \uacb0\uc815\ud558\ub294 ‘\uc815\ucc45’\uc744 \ud559\uc2b5\ud569\ub2c8\ub2e4. \uc815\ucc45\uc740 \uc0c1\ud0dc\ub97c \uc785\ub825\uc73c\ub85c \ubc1b\uc544 \ud589\ub3d9\uc744 \ucd9c\ub825\ud558\ub294 \ud568\uc218\ub85c, \uc8fc\ub85c \uc2e0\uacbd\ub9dd\uc744 \uc0ac\uc6a9\ud558\uc5ec \uad6c\ud604\ub429\ub2c8\ub2e4. \uc815\ucc45 \uae30\ubc18 \ubc29\ubc95\uc758 \uc7a5\uc810\uc740 \uc5f0\uc18d\uc801\uc778 \ud589\ub3d9 \uacf5\uac04\uc744 \ub2e4\ub8f0 \uc218 \uc788\uace0, \ud655\ub960\uc801 \uc815\ucc45\uc744 \uc0ac\uc6a9\ud558\uc5ec \ud0d0\ud5d8\uc744 \uc720\ub3c4\ud560 \uc218 \uc788\ub2e4\ub294 \uc810\uc785\ub2c8\ub2e4.<\/p>\n PPO\ub294 2017\ub144 OpenAI\uc5d0 \uc758\ud574 \uc81c\uc548\ub41c \uc815\ucc45 \uae30\ubc18 \uac15\ud654\ud559\uc2b5 \uc54c\uace0\ub9ac\uc998\uc73c\ub85c, \uc548\uc815\uc131\uacfc \uc0d8\ud50c \ud6a8\uc728\uc131\uc744 \ub3d9\uc2dc\uc5d0 \uc720\uc9c0\ud558\uba74\uc11c\ub3c4 \uc131\ub2a5\uc744 \ub192\uc774\uae30 \uc704\ud574 \uc124\uacc4\ub418\uc5c8\uc2b5\ub2c8\ub2e4. PPO\ub294 \ud074\ub9ac\ud551 \ubc29\uc2dd\uc744 \ud1b5\ud574 \uc815\ucc45 \uc5c5\ub370\uc774\ud2b8\uc758 \ud06c\uae30\ub97c \uc81c\uc5b4\ud558\uc5ec, \uc815\ucc45\uc774 \uc9c0\ub098\uce58\uac8c \ud06c\uac8c \ubcc0\uacbd\ub418\uc9c0 \uc54a\ub3c4\ub85d \ud569\ub2c8\ub2e4.<\/p>\n PPO\ub294 \ub2e4\uc591\ud55c \uc751\uc6a9 \ubd84\uc57c\uc5d0\uc11c \uc131\uacf5\uc801\uc73c\ub85c \uc0ac\uc6a9\ub418\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uadf8 \uc911 \uba87 \uac00\uc9c0\ub97c \uc18c\uac1c\ud558\uaca0\uc2b5\ub2c8\ub2e4:<\/p>\n A3C\ub294 2016\ub144 \uad6c\uae00 \ub525\ub9c8\uc778\ub4dc\uc5d0 \uc758\ud574 \uac1c\ubc1c\ub41c \uac15\ud654\ud559\uc2b5 \uc54c\uace0\ub9ac\uc998\uc73c\ub85c, \uc5ec\ub7ec \uac1c\uc758 \uc5d0\uc774\uc804\ud2b8\ub97c \ube44\ub3d9\uae30\uc801\uc73c\ub85c \ud559\uc2b5\ud558\uc5ec \uc0d8\ud50c \ud6a8\uc728\uc131\uc744 \ub192\uc774\uace0, \uc548\uc815\uc131\uc744 \ud5a5\uc0c1\uc2dc\ud0b5\ub2c8\ub2e4. A3C\ub294 \uac12\uc744 \ud3c9\uac00\ud558\ub294 ‘\uc561\ud130’\uc640 \uc815\ucc45\uc744 \ud559\uc2b5\ud558\ub294 ‘\ud06c\ub9ac\ud2f1’\uc744 \uacb0\ud569\ud55c \uac83\uc785\ub2c8\ub2e4.<\/p>\n A3C\ub294 \ub2e4\uc591\ud55c \ubd84\uc57c\uc5d0\uc11c \uc801\uc6a9\ub418\uace0 \uc788\uc73c\uba70, \uba87 \uac00\uc9c0 \uc8fc\uc694 \uc0ac\ub840\ub294 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4:<\/p>\n Soft Actor-Critic (SAC)\uc740 2018\ub144 \uc81c\uc548\ub41c \uc624\ud504\ub77c\uc778 \uac15\ud654\ud559\uc2b5 \uc54c\uace0\ub9ac\uc998\uc73c\ub85c, \uc561\ud130-\ud06c\ub9ac\ud2f1 \uad6c\uc870\ub97c \uae30\ubc18\uc73c\ub85c \ud558\uba70, \uc5d4\ud2b8\ub85c\ud53c \ubcf4\uc0c1\uc744 \ud3ec\ud568\ud558\uc5ec \uc0d8\ud50c \uc218\uc9d1\uc758 \uc548\uc815\uc131\uc744 \ub192\uc774\uace0, \uc815\ucc45\uc758 \ub2e4\uc591\uc131\uc744 \ubcf4\uc874\ud558\ub294 \ub370 \uc911\uc810\uc744 \ub450\uace0 \uc124\uacc4\ub418\uc5c8\uc2b5\ub2c8\ub2e4.<\/p>\n SAC\ub294 \ubcf5\uc7a1\ud55c \uc5f0\uc18d\uc801 \ud589\ub3d9 \uacf5\uac04\uc744 \uac00\uc9c4 \ub2e4\uc591\ud55c \ubb38\uc81c\uc5d0\uc11c \ub6f0\uc5b4\ub09c \uc131\ub2a5\uc744 \ubcf4\uc785\ub2c8\ub2e4. \ub2e4\uc74c\uc740 \uba87 \uac00\uc9c0 \uc8fc\uc694 \uc0ac\ub840\uc785\ub2c8\ub2e4:<\/p>\n\uc815\ucc45 \uae30\ubc18 \uac15\ud654\ud559\uc2b5\uc758 \uae30\ubcf8 \uac1c\ub150<\/h2>\n
1. Proximal Policy Optimization (PPO)<\/h2>\n
1.1 PPO \uac1c\uc694<\/h3>\n
1.2 PPO\uc758 \ud2b9\uc9d5<\/h3>\n
\n
1.3 PPO \uc54c\uace0\ub9ac\uc998 \uad6c\uc870<\/h3>\n
\n1. \ucd08\uae30\ud654: \uc815\ucc45 \ub124\ud2b8\uc6cc\ud06c\uc640 \uac00\uce58 \ub124\ud2b8\uc6cc\ud06c\ub97c \ucd08\uae30\ud654\ud569\ub2c8\ub2e4.\n2. \ud658\uacbd\uc5d0\uc11c \uc5d0\ud53c\uc18c\ub4dc \uc218\uc9d1: \ud604\uc7ac\uc758 \uc815\ucc45\uc744 \uae30\ubc18\uc73c\ub85c \ud658\uacbd\uacfc \uc0c1\ud638\uc791\uc6a9\ud558\uc5ec \uc0d8\ud50c\uc744 \uc218\uc9d1\ud569\ub2c8\ub2e4.\n3. \ubcf4\uc0c1 \uacc4\uc0b0: \uc218\uc9d1\ub41c \uc0d8\ud50c\uc5d0 \ub300\ud574 \ud560\uc778\ub41c \ubcf4\uc0c1\uc744 \uacc4\uc0b0\ud569\ub2c8\ub2e4.\n4. \uc815\ucc45 \uc5c5\ub370\uc774\ud2b8:\n a. \ud074\ub9ac\ud551 \uc190\uc2e4 \ud568\uc218\ub97c \uc815\uc758\ud569\ub2c8\ub2e4.\n b. \uc190\uc2e4 \ud568\uc218\ub97c \ucd5c\uc18c\ud654\ud558\ub294 \ubc29\ud5a5\uc73c\ub85c \uc815\ucc45 \ud30c\ub77c\ubbf8\ud130\ub97c \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.\n5. \ubc18\ubcf5: 2-4 \ub2e8\uacc4\ub97c \ubc18\ubcf5\ud558\uc5ec \uc815\ucc45\uc744 \uac1c\uc120\ud569\ub2c8\ub2e4.\n<\/code><\/pre>\n1.4 PPO \uad6c\ud604 \uc608\uc81c<\/h3>\n
\nimport gym\nimport numpy as np\nimport tensorflow as tf\n\nclass PPOAgent:\n def __init__(self, state_size, action_size):\n self.state_size = state_size\n self.action_size = action_size\n self.policy_model = self.build_model()\n self.value_model = self.build_value_model()\n\n def build_model(self):\n model = tf.keras.Sequential([\n tf.keras.layers.Dense(24, activation='relu', input_shape=(self.state_size,)),\n tf.keras.layers.Dense(24, activation='relu'),\n tf.keras.layers.Dense(self.action_size, activation='softmax')\n ])\n model.compile(optimizer='adam', loss='categorical_crossentropy')\n return model\n\n def build_value_model(self):\n model = tf.keras.Sequential([\n tf.keras.layers.Dense(24, activation='relu', input_shape=(self.state_size,)),\n tf.keras.layers.Dense(24, activation='relu'),\n tf.keras.layers.Dense(1)\n ])\n model.compile(optimizer='adam', loss='mean_squared_error')\n return model\n\n def update_policy(self, states, actions, advantages):\n # \uc815\ucc45 \uc5c5\ub370\uc774\ud2b8 \ucf54\ub4dc\n pass # \uc774\uacf3\uc5d0 PPO\uc758 \uc5c5\ub370\uc774\ud2b8 \ub85c\uc9c1\uc744 \uc0bd\uc785\ud569\ub2c8\ub2e4.\n\n# \ud658\uacbd\uacfc \uc5d0\uc774\uc804\ud2b8\ub97c \ucd08\uae30\ud654\nenv = gym.make('CartPole-v1')\nagent = PPOAgent(env.observation_space.shape[0], env.action_space.n)\n<\/code><\/pre>\n1.5 PPO\uc758 \uc751\uc6a9 \uc0ac\ub840<\/h3>\n
\n
2. Asynchronous Actor-Critic (A3C)<\/h2>\n
2.1 A3C \uac1c\uc694<\/h3>\n
2.2 A3C\uc758 \ud2b9\uc9d5<\/h3>\n
\n
2.3 A3C \uc54c\uace0\ub9ac\uc998 \uad6c\uc870<\/h3>\n
\n1. \uc5ec\ub7ec \uac1c\uc758 \ud074\ub860 \uc5d0\uc774\uc804\ud2b8\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4.\n2. \uac01 \uc5d0\uc774\uc804\ud2b8\uac00 \ud658\uacbd\uc5d0\uc11c \uacbd\ud5d8\uc744 \uc218\uc9d1\ud569\ub2c8\ub2e4.\n3. \uac01 \uc5d0\uc774\uc804\ud2b8\uc758 \uacbd\ud5d8\uc744 \uacf5\uc720\ud558\uc5ec \uacf5\ud1b5\uc758 \uae00\ub85c\ubc8c \ubaa8\ub378\uc744 \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.\n4. \uae00\ub85c\ubc8c \ubaa8\ub378\uc758 \ud30c\ub77c\ubbf8\ud130\ub97c \uac01 \uc5d0\uc774\uc804\ud2b8\uc5d0 \uc804\uc1a1\ud558\uc5ec \ub3d9\uae30\ud654\ud569\ub2c8\ub2e4.\n5. 2-4 \ub2e8\uacc4\ub97c \ubc18\ubcf5\ud558\uc5ec \ud559\uc2b5\ud569\ub2c8\ub2e4.\n<\/code><\/pre>\n2.4 A3C \uad6c\ud604 \uc608\uc81c<\/h3>\n
\nimport tensorflow as tf\nimport gym\nimport threading\nimport numpy as np\n\nclass A3CAgent:\n def __init__(self, state_size, action_size):\n self.global_model = self.build_model()\n self.optimizer = tf.keras.optimizers.Adam()\n\n def build_model(self):\n model = tf.keras.Sequential([\n tf.keras.layers.Dense(24, activation='relu'),\n tf.keras.layers.Dense(24, activation='relu'),\n tf.keras.layers.Dense(action_size, activation='softmax')\n ])\n return model\n\n def train(self):\n pass # A3C\uc758 \ud559\uc2b5 \ub85c\uc9c1\uc744 \uad6c\ud604\ud569\ub2c8\ub2e4.\n\n# \ud658\uacbd\uacfc \uc5d0\uc774\uc804\ud2b8 \uc0dd\uc131\nenv = gym.make('CartPole-v1')\nagent = A3CAgent(env.observation_space.shape[0], env.action_space.n)\n<\/code><\/pre>\n2.5 A3C\uc758 \uc751\uc6a9 \uc0ac\ub840<\/h3>\n
\n
3. Soft Actor-Critic (SAC)<\/h2>\n
3.1 SAC \uac1c\uc694<\/h3>\n
3.2 SAC\uc758 \ud2b9\uc9d5<\/h3>\n
\n
3.3 SAC \uc54c\uace0\ub9ac\uc998 \uad6c\uc870<\/h3>\n
\n1. \uc815\ucc45 \ub124\ud2b8\uc6cc\ud06c\uc640 \ub450 \uac1c\uc758 \uac00\uce58 \ub124\ud2b8\uc6cc\ud06c\ub97c \ucd08\uae30\ud654\ud569\ub2c8\ub2e4.\n2. \uacbd\ud5d8 \ub9ac\ud50c\ub808\uc774 \ubc84\ud37c\ub97c \uc0ac\uc6a9\ud558\uc5ec \uc0d8\ud50c\uc744 \uc800\uc7a5\ud569\ub2c8\ub2e4.\n3. \uc5d0\uc774\uc804\ud2b8\uac00 \ud658\uacbd\uacfc \uc0c1\ud638\uc791\uc6a9\ud558\uc5ec \uacbd\ud5d8\uc744 \uc218\uc9d1\ud569\ub2c8\ub2e4.\n4. \uc5d4\ud2b8\ub85c\ud53c \ubcf4\uc0c1\uc744 \ud3ec\ud568\ud55c \uc190\uc2e4 \ud568\uc218\ub97c \uae30\ubc18\uc73c\ub85c \ub124\ud2b8\uc6cc\ud06c\ub97c \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.\n5. \ubc18\ubcf5\ud558\uc5ec \uc815\ucc45\uc758 \uc131\ub2a5\uc744 \ud5a5\uc0c1\uc2dc\ud0b5\ub2c8\ub2e4.\n<\/code><\/pre>\n3.4 SAC \uad6c\ud604 \uc608\uc81c<\/h3>\n
\nimport tensorflow as tf\nimport gym\n\nclass SACAgent:\n def __init__(self, state_size, action_size):\n self.policy_model = self.build_policy_model()\n self.value_model1 = self.build_value_model()\n self.value_model2 = self.build_value_model()\n\n def build_policy_model(self):\n model = tf.keras.Sequential([\n tf.keras.layers.Dense(256, activation='relu'),\n tf.keras.layers.Dense(256, activation='relu'),\n tf.keras.layers.Dense(action_size, activation='tanh')\n ])\n return model\n\n def build_value_model(self):\n model = tf.keras.Sequential([\n tf.keras.layers.Dense(256, activation='relu'),\n tf.keras.layers.Dense(256, activation='relu'),\n tf.keras.layers.Dense(1)\n ])\n return model\n\n def update(self):\n pass # SAC\uc758 \uc5c5\ub370\uc774\ud2b8 \ub85c\uc9c1\uc744 \uad6c\ud604\ud569\ub2c8\ub2e4.\n\n# \ud658\uacbd\uacfc \uc5d0\uc774\uc804\ud2b8 \uc0dd\uc131\nenv = gym.make('Pendulum-v0')\nagent = SACAgent(env.observation_space.shape[0], env.action_space.shape[0])\n<\/code><\/pre>\n3.5 SAC\uc758 \uc751\uc6a9 \uc0ac\ub840<\/h3>\n