\uac15\ud654\ud559\uc2b5(Reinforcement Learning, RL)\uc740 \uc5d0\uc774\uc804\ud2b8\uac00 \ud658\uacbd\uacfc \uc0c1\ud638\uc791\uc6a9\ud558\uc5ec \ubcf4\uc0c1\uc744 \uadf9\ub300\ud654\ud558\ub294 \ubc29\ubc95\uc744 \ubc30\uc6b0\ub294 \uae30\uacc4 \ud559\uc2b5\uc758 \ud55c \ubd84\uc57c\uc785\ub2c8\ub2e4. \uc774\ubc88 \uae00\uc5d0\uc11c\ub294 \uae30\ubcf8\uc801\uc778 \uac15\ud654\ud559\uc2b5 \uc54c\uace0\ub9ac\uc998\uc778 Q-\ub7ec\ub2dd\uacfc SARSA\ub97c \uc18c\uac1c\ud558\uace0, \uc774 \ub450 \uc54c\uace0\ub9ac\uc998\uc758 \ucc28\uc774\uc810\uc744 \uc0b4\ud3b4\ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \ub610\ud55c, \uac04\ub2e8\ud55c \uc608\uc81c\ub97c \ud1b5\ud574 \uc774 \uc54c\uace0\ub9ac\uc998\ub4e4\uc774 \uc5b4\ub5bb\uac8c \uc791\ub3d9\ud558\ub294\uc9c0\ub97c \uc790\uc138\ud788 \uc54c\uc544\ubcf4\uaca0\uc2b5\ub2c8\ub2e4.<\/p>\n
\uac15\ud654\ud559\uc2b5\uc740 \ub2e4\uc74c\uacfc \uac19\uc740 \uad6c\uc131 \uc694\uc18c\ub85c \uc774\ub8e8\uc5b4\uc838 \uc788\uc2b5\ub2c8\ub2e4:<\/p>\n
Q-\ub7ec\ub2dd\uc740 \ubaa8\ub378 \ud504\ub9ac(model-free) \uac15\ud654\ud559\uc2b5 \uc54c\uace0\ub9ac\uc998\uc73c\ub85c, \uc8fc\uc5b4\uc9c4 \uc0c1\ud0dc\uc5d0\uc11c \ucd5c\uc801\uc758 \ud589\ub3d9\uc744 \uc120\ud0dd\ud558\uae30 \uc704\ud574 Q-\uac12(Q-value)\uc744 \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4. Q-\uac12\uc740 \ud2b9\uc815 \uc0c1\ud0dc\uc5d0\uc11c \ud2b9\uc815 \ud589\ub3d9\uc744 \ucde8\ud588\uc744 \ub54c \uae30\ub300\ub418\ub294 \uc7a5\uae30\uc801\uc778 \ubcf4\uc0c1\uc758 \ucd94\uc815\uce58\uc785\ub2c8\ub2e4.<\/p>\n
Q-\ub7ec\ub2dd\uc758 \ud575\uc2ec\uc740 \ub2e4\uc74c\uc758 Q-\uac12 \uc5c5\ub370\uc774\ud2b8 \ubc29\uc815\uc2dd\uc785\ub2c8\ub2e4:<\/p>\n
\nQ(s, a) <- Q(s, a) + \u03b1 * (r + \u03b3 * max_a’ Q(s’, a’) – Q(s, a))<\/p>\n<\/blockquote>\n
\n
- s<\/strong>: \ud604\uc7ac \uc0c1\ud0dc<\/li>\n
- a<\/strong>: \ud604\uc7ac \ud589\ub3d9<\/li>\n
- r<\/strong>: \ud604\uc7ac \uc0c1\ud0dc\uc5d0\uc11c \ucde8\ud55c \ud589\ub3d9\uc5d0 \ub300\ud55c \ubcf4\uc0c1<\/li>\n
- s’<\/strong>: \ub2e4\uc74c \uc0c1\ud0dc<\/li>\n
- \u03b1<\/strong>: \ud559\uc2b5\ub960 (0 < \u03b1 < 1)<\/li>\n
- \u03b3<\/strong>: \ud560\uc778\uc728 (0 < \u03b3 < 1)<\/li>\n
- max_a’<\/strong>: \ub2e4\uc74c \uc0c1\ud0dc\uc5d0\uc11c \uac00\ub2a5\ud55c \ud589\ub3d9 \uc911 \ucd5c\ub300 Q-\uac12\uc744 \uc120\ud0dd<\/li>\n<\/ul>\n
2.2. Q-\ub7ec\ub2dd\uc758 \uc791\ub3d9 \ubc29\uc2dd<\/h3>\n
Q-\ub7ec\ub2dd \uc54c\uace0\ub9ac\uc998\uc740 \ub2e4\uc74c\uacfc \uac19\uc740 \uc808\ucc28\ub85c \uc9c4\ud589\ub429\ub2c8\ub2e4:<\/p>\n
\n
- \ucd08\uae30 Q-\uac12\uc744 \ubb34\uc791\uc704\ub85c \uc124\uc815\ud569\ub2c8\ub2e4.<\/li>\n
- \uc5d0\uc774\uc804\ud2b8\ub97c \ucd08\uae30 \uc0c1\ud0dc\ub85c \uc124\uc815\ud569\ub2c8\ub2e4.<\/li>\n
- \uc885\ub8cc \uc870\uac74\uc774 \ub9cc\uc871\ub420 \ub54c\uae4c\uc9c0 \ubc18\ubcf5\ud569\ub2c8\ub2e4:<\/li>\n
\n
- \ud604\uc7ac \uc0c1\ud0dc\uc5d0\uc11c \ud589\ub3d9\uc744 \uc120\ud0dd\ud569\ub2c8\ub2e4 (\ud0d0\uc0c9 \ub610\ub294 \ud65c\uc6a9).<\/li>\n
- \ud589\ub3d9\uc744 \uc218\ud589\ud558\uace0, \ubcf4\uc0c1\uacfc \ub2e4\uc74c \uc0c1\ud0dc\ub97c \uad00\ucc30\ud569\ub2c8\ub2e4.<\/li>\n
- Q-\uac12\uc744 \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.<\/li>\n
- \ud604\uc7ac \uc0c1\ud0dc\ub97c \ub2e4\uc74c \uc0c1\ud0dc\ub85c \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.<\/li>\n<\/ul>\n<\/ol>\n
2.3. Q-\ub7ec\ub2dd \uac04\ub2e8\ud55c \uc608\uc81c<\/h3>\n
\ub2e4\uc74c\uc740 OpenAI\uc758 Gym \ub77c\uc774\ube0c\ub7ec\ub9ac\uc5d0\uc11c \uc81c\uacf5\ud558\ub294 ‘Taxi-v3’ \ud658\uacbd\uc5d0\uc11c Q-\ub7ec\ub2dd\uc744 \uad6c\ud604\ud55c \uac04\ub2e8\ud55c \uc608\uc81c\uc785\ub2c8\ub2e4:<\/p>\n
\n\nimport numpy as np\nimport gym\n\n# \ud658\uacbd \ub9cc\ub4e4\uae30\nenv = gym.make('Taxi-v3')\n\n# Q-\ud14c\uc774\ube14 \ucd08\uae30\ud654\nQ = np.zeros((env.observation_space.n, env.action_space.n))\n\n# \ud558\uc774\ud37c\ud30c\ub77c\ubbf8\ud130\nalpha = 0.1 # \ud559\uc2b5\ub960\ngamma = 0.6 # \ud560\uc778\uc728\nepsilon = 0.1 # \ud0d0\uc0c9 \ud655\ub960\n\n# \uc5d0\ud53c\uc18c\ub4dc \uc218\nnum_episodes = 1000\n\n# Q-\ub7ec\ub2dd \uc54c\uace0\ub9ac\uc998\nfor i in range(num_episodes):\n state = env.reset()\n done = False\n \n while not done:\n # \ud589\ub3d9 \uc120\ud0dd\n if np.random.rand() < epsilon:\n action = env.action_space.sample() # \ud0d0\uc0c9\n else:\n action = np.argmax(Q[state]) # \ud65c\uc6a9\n \n next_state, reward, done, info = env.step(action)\n \n # Q-\uac12 \uc5c5\ub370\uc774\ud2b8\n Q[state, action] += alpha * (reward + gamma * np.max(Q[next_state]) - Q[state, action])\n \n state = next_state\n\nprint(\"\ud559\uc2b5\ub41c Q-\ud14c\uc774\ube14:\")\nprint(Q)\n <\/code>\n <\/pre>\n3. SARSA<\/h2>\n
SARSA(State-Action-Reward-State-Action)\ub294 Q-\ub7ec\ub2dd\uacfc \uc720\uc0ac\ud55c \ub610 \ub2e4\ub978 \uac15\ud654\ud559\uc2b5 \uc54c\uace0\ub9ac\uc998\uc785\ub2c8\ub2e4. SARSA\uc758 \ucc28\ubcc4\uc810\uc740 Q-\uac12\uc744 \uc5c5\ub370\uc774\ud2b8\ud560 \ub54c \ucd5c\uc801\uc758 \ud589\ub3d9\uc774 \uc544\ub2cc \uc2e4\uc81c \ucde8\ud55c \ud589\ub3d9(action)\uc744 \uc0ac\uc6a9\ud55c\ub2e4\ub294 \uac83\uc785\ub2c8\ub2e4. \uc774\ub7ec\ud55c \ubc29\uc2dd\uc740 \uc5d0\uc774\uc804\ud2b8\uc758 \ud0d0\uc0c9 \uacbd\ub85c\uc5d0 \ub354 \uc758\uc874\ud558\uba70, \uc880 \ub354 \ubcf4\uc218\uc801\uc778 \uc815\ucc45\uc744 \ud615\uc131\ud569\ub2c8\ub2e4.<\/p>\n
3.1. SARSA \uc54c\uace0\ub9ac\uc998 \uc2dd<\/h3>\n
SARSA\uc758 Q-\uac12 \uc5c5\ub370\uc774\ud2b8 \ubc29\uc815\uc2dd\uc740 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4:<\/p>\n
\nQ(s, a) <- Q(s, a) + \u03b1 * (r + \u03b3 * Q(s’, a’) – Q(s, a))<\/p>\n<\/blockquote>\n
\n
- \uac01 \uc694\uc18c\uc758 \uc758\ubbf8\ub294 \uc55e\uc11c \uc124\uba85\ud55c Q-\ub7ec\ub2dd\uacfc \uc720\uc0ac\ud569\ub2c8\ub2e4. \ub2e8, \uc5ec\uae30\uc11c \uc0ac\uc6a9\ub418\ub294 a’\ub294 \ub2e4\uc74c \uc0c1\ud0dc s’\uc5d0\uc11c \uc2e4\uc81c\ub85c \uc120\ud0dd\ub41c \ud589\ub3d9\uc785\ub2c8\ub2e4.<\/li>\n<\/ul>\n
3.2. SARSA\uc758 \uc791\ub3d9 \ubc29\uc2dd<\/h3>\n
SARSA \uc54c\uace0\ub9ac\uc998\uc740 \ub2e4\uc74c\uacfc \uac19\uc740 \uc808\ucc28\ub85c \uc9c4\ud589\ub429\ub2c8\ub2e4:<\/p>\n
\n
- \ucd08\uae30 Q-\uac12\uc744 \ubb34\uc791\uc704\ub85c \uc124\uc815\ud569\ub2c8\ub2e4.<\/li>\n
- \uc5d0\uc774\uc804\ud2b8\ub97c \ucd08\uae30 \uc0c1\ud0dc\ub85c \uc124\uc815\ud569\ub2c8\ub2e4.<\/li>\n
- \uc885\ub8cc \uc870\uac74\uc774 \ub9cc\uc871\ub420 \ub54c\uae4c\uc9c0 \ubc18\ubcf5\ud569\ub2c8\ub2e4:<\/li>\n
\n
- \ud604\uc7ac \uc0c1\ud0dc\uc5d0\uc11c \ud589\ub3d9\uc744 \uc120\ud0dd\ud569\ub2c8\ub2e4.<\/li>\n
- \ud589\ub3d9\uc744 \uc218\ud589\ud558\uace0, \ubcf4\uc0c1\uacfc \ub2e4\uc74c \uc0c1\ud0dc\ub97c \uad00\ucc30\ud569\ub2c8\ub2e4.<\/li>\n
- \ub2e4\uc74c \uc0c1\ud0dc\uc5d0\uc11c \ud589\ub3d9\uc744 \uc120\ud0dd\ud569\ub2c8\ub2e4.<\/li>\n
- Q-\uac12\uc744 \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.<\/li>\n
- \ud604\uc7ac \uc0c1\ud0dc\uc640 \ud589\ub3d9\uc744 \ub2e4\uc74c \uc0c1\ud0dc\uc640 \ud589\ub3d9\uc73c\ub85c \uc5c5\ub370\uc774\ud2b8\ud569\ub2c8\ub2e4.<\/li>\n<\/ul>\n<\/ol>\n
3.3. SARSA \uac04\ub2e8\ud55c \uc608\uc81c<\/h3>\n
\ub2e4\uc74c\uc740 ‘Taxi-v3’ \ud658\uacbd\uc5d0\uc11c SARSA \uc54c\uace0\ub9ac\uc998\uc744 \uad6c\ud604\ud55c \uac04\ub2e8\ud55c \uc608\uc81c\uc785\ub2c8\ub2e4:<\/p>\n
\n\nimport numpy as np\nimport gym\n\n# \ud658\uacbd \ub9cc\ub4e4\uae30\nenv = gym.make('Taxi-v3')\n\n# Q-\ud14c\uc774\ube14 \ucd08\uae30\ud654\nQ = np.zeros((env.observation_space.n, env.action_space.n))\n\n# \ud558\uc774\ud37c\ud30c\ub77c\ubbf8\ud130\nalpha = 0.1 # \ud559\uc2b5\ub960\ngamma = 0.6 # \ud560\uc778\uc728\nepsilon = 0.1 # \ud0d0\uc0c9 \ud655\ub960\n\n# \uc5d0\ud53c\uc18c\ub4dc \uc218\nnum_episodes = 1000\n\n# SARSA \uc54c\uace0\ub9ac\uc998\nfor i in range(num_episodes):\n state = env.reset()\n action = env.action_space.sample() if np.random.rand() < epsilon else np.argmax(Q[state])\n done = False\n \n while not done:\n next_state, reward, done, info = env.step(action)\n next_action = env.action_space.sample() if np.random.rand() < epsilon else np.argmax(Q[next_state])\n \n # Q-\uac12 \uc5c5\ub370\uc774\ud2b8\n Q[state, action] += alpha * (reward + gamma * Q[next_state, next_action] - Q[state, action])\n \n state, action = next_state, next_action\n\nprint(\"\ud559\uc2b5\ub41c Q-\ud14c\uc774\ube14:\")\nprint(Q)\n <\/code>\n <\/pre>\n4. Q-\ub7ec\ub2dd\uacfc SARSA\uc758 \ucc28\uc774\uc810<\/h2>\n
Q-\ub7ec\ub2dd\uacfc SARSA\uc758 \uc8fc\uc694 \ucc28\uc774\uc810\uc740 Q-\uac12 \uc5c5\ub370\uc774\ud2b8 \ubc29\uc2dd\uc785\ub2c8\ub2e4. Q-\ub7ec\ub2dd\uc740 \uc8fc\uc5b4\uc9c4 \ub2e4\uc74c \uc0c1\ud0dc\uc758 \ucd5c\uc801 \ud589\ub3d9\uc744 \uc0ac\uc6a9\ud558\uc5ec Q-\uac12\uc744 \uc5c5\ub370\uc774\ud2b8\ud558\ub294 \ubc18\uba74, SARSA\ub294 \uc2e4\uc81c\ub85c \ucde8\ud55c \ud589\ub3d9\uc744 \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \uc774\ub7ec\ud55c \ucc28\uc774\ub294 \ub2e4\uc74c\uacfc \uac19\uc740 \uacb0\uacfc\ub97c \ucd08\ub798\ud569\ub2c8\ub2e4:<\/p>\n
\n
- Q-\ub7ec\ub2dd<\/strong>: \ucd5c\uc801\uc758 \ud589\ub3d9\uc744 \ucd94\uc815\ud558\uae30 \ub54c\ubb38\uc5d0 \ud559\uc2b5 \uc18d\ub3c4\uac00 \ub354 \ube60\ub97c \uc218 \uc788\uc2b5\ub2c8\ub2e4.<\/li>\n
- SARSA<\/strong>: \uc2e4\uc81c\ub85c \ucde8\ud55c \ud589\ub3d9\uc744 \uae30\ubc18\uc73c\ub85c \ud558\uc5ec \ub354 \ubcf4\uc218\uc801\uc778 \uc815\ucc45\uc744 \ucde8\ud569\ub2c8\ub2e4.<\/li>\n<\/ul>\n
5. \ub9c8\ubb34\ub9ac<\/h2>\n
\uc774\ubc88 \uae00\uc5d0\uc11c\ub294 \uac15\ud654\ud559\uc2b5\uc758 \uae30\ubcf8 \uc54c\uace0\ub9ac\uc998\uc778 Q-\ub7ec\ub2dd\uacfc SARSA\ub97c \uc18c\uac1c\ud558\uc600\uc2b5\ub2c8\ub2e4. \ub450 \uc54c\uace0\ub9ac\uc998\uc758 \uae30\ubcf8 \uc6d0\ub9ac\uc640 \ucc28\uc774\uc810\uc744 \uc0b4\ud3b4\ubcf4\uc558\uc73c\uba70, \uac01\uac01\uc758 \uad6c\ud604 \uc608\uc81c\ub97c \ud1b5\ud574 \uc774\ud574\ub97c \ub3d5\uace0\uc790 \ud558\uc600\uc2b5\ub2c8\ub2e4. \uc774\ub7ec\ud55c \uae30\ubcf8 \uc54c\uace0\ub9ac\uc998\ub4e4\uc740 \uac15\ud654\ud559\uc2b5\uc758 \uae30\ubc18\uc774 \ub418\uba70, \ub354 \ubcf5\uc7a1\ud55c \uc54c\uace0\ub9ac\uc998\uc73c\ub85c \ubc1c\uc804\ud558\ub294 \ub370\uc5d0 \uc911\uc694\ud55c \uae30\ucd08 \uc9c0\uc2dd\uc744 \uc81c\uacf5\ud569\ub2c8\ub2e4.<\/p>\n
\uac15\ud654\ud559\uc2b5\uc740 \ub2e4\uc591\ud55c \ubd84\uc57c\uc5d0 \uc751\uc6a9\ub420 \uc218 \uc788\uc73c\uba70, \uc774\ub97c \ud1b5\ud574 \ub354\uc6b1 \ubc1c\uc804\ud55c \uc54c\uace0\ub9ac\uc998\uacfc \uae30\uc220\ub4e4\uc774 \uc9c0\uae08\ub3c4 \uc5f0\uad6c\ub418\uace0 \uc788\uc2b5\ub2c8\ub2e4. \uc55e\uc73c\ub85c\ub3c4 \uac15\ud654\ud559\uc2b5\uc5d0 \ub300\ud55c \uad00\uc2ec\uc744 \uac00\uc9c0\uace0 \uc9c0\uc18d\uc801\uc778 \ud559\uc2b5\uc744 \uc774\uc5b4\ub098\uac00\uc2dc\uae38 \ubc14\ub78d\ub2c8\ub2e4.<\/p>\n
<\/body><\/p>\n","protected":false},"excerpt":{"rendered":"