A Painless Q-Learning Tutorial

May 16, 2015 03:41 PM  well written; very clear;

doors to the outside get an instant reward of 100 (doors 1 and 4)

room 5 = the outside... an ABSORBING state:  agent stays there forever (it loops back on itself)

See the Matrix R  (the reward matrix)

Below the 6 by 6 matrix R of states and actions:

Now we'll add a similar matrix, "Q", to the brain of our agent,

representing the memory of what the agent has learned through experience.

The rows of matrix Q represent the current state of the agent, and the columns represent the possible actions leading to the next state (the links between the nodes).

*

Q matrix is initially ZERO.  (can start with no rows or coumns; then just add them as we discover new ones.)

how to update elements in the matrix:

The transition rule of Q learning is a very simple formula:

Q(state, action) = R(state, action) + Gamma * Max[Q(next state, all actions)]

</blockquote>

According to this formula, a value assigned to a specific element of matrix Q, is equal to the sum of the corresponding value in matrix R and the learning parameter Gamma, multiplied by the

maximum value of Q for all possible actions in the next state.

(that is, chose that action which maximizes the reward over all possible actions!!!)  YES!

*

look at the pseudo-code computer program:  the crux step is...

<ul >

<li >Select one among all possible actions for the current state.</li>

<li >Using this possible action, consider going to the next state.</li>

<li >Get maximum Q value for this next state based on all possible actions.</li>

<li >Compute: Q(state, action) = R(state, action) + Gamma * Max[Q(next state, all actions)]</li>

</ul>

A Painless Q-Learning Tutorial

Attachments