Deep Learning PyTorch Course, Q-Learning

1. What is Q-Learning?

Q-Learning is a form of reinforcement learning that helps an agent learn the optimal behaviors by interacting with the environment. The core idea of Q-Learning is to use a queue that stores the values for possible actions in each state. This aids the agent in determining the optimal action it can take.

Q-Learning is generally based on the Markov Decision Process (MDP) and is composed of the following elements:

• State (S): The situation the agent is in within the environment.
• Action (A): The possible actions the agent can take.
• Reward (R): The score the agent receives for taking a specific action.
• Value Function (Q): A measure of how good a particular action is in a given state.

2. Q-Learning Algorithm

The Q-Learning algorithm includes the basic idea of updating the Q function. The agent follows the procedure outlined below at each time step:

1. Select an action based on the current state.
2. Observe the new state and receive a reward after performing the selected action.
3. Update the Q function.

The Q function update can be expressed using the following formula:

Q(S, A) <- Q(S, A) + α(R + γ * max(Q(S', A')) - Q(S, A))

Here, α represents the learning rate, and γ denotes the discount factor. These two elements determine how much the agent reflects on past experiences.

3. Implementing Q-Learning with PyTorch

Now, let’s implement Q-Learning simply using PyTorch. In this example, we will create an environment using OpenAI’s Gym library and train a Q-Learning agent.

import gym
import numpy as np
import random

# Hyperparameters
LEARNING_RATE = 0.1
DISCOUNT_FACTOR = 0.9
EPISODES = 1000

# Environment setup
env = gym.make('Taxi-v3')
Q_table = np.zeros([env.observation_space.n, env.action_space.n])

def select_action(state, epsilon):
    if random.uniform(0, 1) < epsilon:
        return env.action_space.sample()  # Select random action
    else:
        return np.argmax(Q_table[state])  # Select action with the highest Q value

for episode in range(EPISODES):
    state = env.reset()
    done = False
    epsilon = 1.0 / (episode / 100 + 1)  # Exploration rate

    while not done:
        action = select_action(state, epsilon)
        next_state, reward, done, _ = env.step(action)
        
        # Update Q function
        Q_table[state][action] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * np.max(Q_table[next_state]) - Q_table[state][action])
        
        state = next_state

print("Training Complete")

# Sample Test
state = env.reset()
done = False
while not done:
    action = np.argmax(Q_table[state])  # Select optimal action
    state, reward, done, _ = env.step(action)
    env.render()  # Render environment

4. Advantages and Disadvantages of Q-Learning

The main advantages of Q-Learning are:

• A simple and easy-to-understand algorithm
• Operates well in model-free environments

However, it has the following disadvantages:

• Learning speed may decrease when the state space is large
• The exploration-exploitation balance can be challenging