Skip to main content


Showing posts from 2009

Temporal-Difference Learning Policy Evaluation in Python

In the code bellow, is an example of policy evaluation for very simple task. Example is taken from the book: "Reinforcement Learning: An Introduction, Surto and Barto".

This is an example of policy evaluation for a random walk policy.

Example 6.2: Random Walk from the book:
"Reinforcement Learning: An Introduction, Surto and Barto"

The policy is evaluated by dynamic programing and TD(0).

In this example, the policy can start in five states 1, 2, 3, 4, 5 and end in
two states 0 and 6. The allowed transitions between the states are as follwes:

0 <-> 1 <-> 2 <-> 3 <-> 4 <-> 5 <-> 6

The reward for ending in the state 6 is 1.
The reward for ending in the state 0 is 0.

In any state, except the final states, you can take two actions: 'left' and 'right'.
In the final states the policy and episodes end.

Because this example implements the random walk policy then both actions can be
taken with th…

Difference between the Standard Deviation, Standard Error and Confidence Itervals

In its simplest: the standard deviation represents the variability of input values,
the standard error (of the mean) represents variability of computed mean,
the confidence intervals represents where the 'true' mean value might lie.Computation: the standard deviation is computed from the variance of your data - input values,
the standard error is computed from the standard deviation,
the confidence intervals are computed from the standard error. Read this:
Many people confuse the standard deviation (SD) and the standard error of the mean (SE) and are unsure which, if either, to use in presenting data in graphical or tabular form. The SD is an index of the variability of the original data points and should be reported in all studies. The SE reflects the variability of the mean values, as if the study were repeated a large number of times. By itself, the SE is not particularly useful; however, it is used in constructing 95% and 99% confidence intervals (CIs), which indicate a range of v…

Google Testing Blog: TotT: EXPECT vs. ASSERT

Google commented on different types of assertations in their testing framework (Google Testing Blog: EXPECT vs. ASSERT)
I have found assertations in my code very useful on many occasions; however, I do not see any need for the EXPECT function. If your code is broken then it is broken and there is no point in continuing and testing conditions which are not likely to be met. 
It is like with C++ compiler errors. The most important error is the firtst error. The rest of the erorrs is usually rubish and useless.

Linked list sorted in an ascending order

Again to practice some of skills I wrote linked list sorted in an ascending order. It is very simple example and it performs only two operation:inserting an item into the appropriate position in the listprinting the whole list on the screenThe following code is placed in the public domain. Use it as you wish. The code can be downloaded with the solution project file for MS Visual Studio 2008 from here.
// LinkedList.cpp : Defines the entry point for the console application.

#include "stdafx.h"

The supplied C code fragment is intended to implement a linked list of
integers stored in ascending order. Each element of the list is a struct
of type Item holding the integer value, a pointer pred to the previous
element, if any, and a pointer succ to the succeeding element, if any. The
variable head points to the ?rst element in the list, and tail points to
the last element. Initially, both head and tail are NULL. The function
insert is intended to insert its argument x into the …

Viterbi Algorithm in C++ and using STL

To practice my C++ and STL skills, I implemented the Viterbi algorithm example from the Wikipedia page: The original algorithm was implemented in Python. I reimplemented the example in C++ and I used STL (mainly vector and map classes).  This code is in public-domain. So, use it as you want. 
The complete solution for MS Visual C++ 2008 can be found at

// ViterbiSTL.cpp : is an C++ and STL implementatiton of the Wikipedia example // Wikipedia:
// It as accurate implementation as it was possible

#include "stdafx.h"
#include "string" #include "vector" #include "map" #include "iostream"
using namespace std;
//states = ('Rainy', 'Sunny') //  //observations = ('walk', 'shop', 'clean') //  //start_probability = {'Rainy': 0.6, 'Sunny': 0.…