Question

1. Answer the following about reinforcement learning:

a. Explain how the e-greedy approach balances exploration and exploitation.
b. Explain how the incremental mean method in the utility mean update reduces the memory usage.
c. Explain the difference between the Monte-Carlo (MC) method and the Temporal Difference (TD) method.
d. Explain how the temporal-difference (TD) method in the utility update reduces the memory usage.
e. Explain in what occasion the TD Q-value update might result different results under SARSA learning and Q-Learning, respectively.

Answer 1

The e-greedy approach balances exploration and exploitation, the incremental mean method reduces memory usage, the MC method and the TD method have different ways of estimating utility, the TD method reduces memory usage, and the TD Q-value update can result in different results under SARSA learning and Q-Learning.

Step-by-step explanation:

1. a. The e-greedy approach balances exploration and exploitation by randomly exploring new actions a certain percentage of the time (epsilon value) and choosing the action with the highest expected reward the rest of the time. This allows the agent to try different actions and learn which ones result in higher rewards.
2. b. The incremental mean method updates the utility values by calculating the average of the current utility and the new observation. This reduces memory usage because it does not need to store all previous observations, only the current utility estimate.
3. c. The Monte-Carlo (MC) method estimates the utility based on the average of the total rewards obtained from complete episodes. The Temporal Difference (TD) method estimates the utility by bootstrapping, using the current estimate and the estimated utility of the next state.
4. d. The temporal-difference (TD) method reduces memory usage by updating the utility values iteratively, using the current estimate and the estimated utility of the next state, instead of waiting for the end of an episode to update all utility values.
5. e. The TD Q-value update can result in different results under SARSA learning and Q-Learning. SARSA learning updates the Q-values based on the current action and the next action chosen by the policy. Q-Learning updates the Q-values based on the current action and the action with the highest Q-value in the next state. This can lead to differences in the learned policy and can affect the convergence and optimal policy achieved.