Monday, December 14, 2009

Information Theory, Inference, and Learning Algorithms

If you are looking for a very good book about machine learning look at this free on-line
Information Theory, Inference, and Learning Algorithms book.

Thursday, April 02, 2009

Temporal-Difference Learning Policy Evaluation in Python

In the code bellow, is an example of policy evaluation for very simple task. Example is taken from the book: "Reinforcement Learning: An Introduction, Surto and Barto".

#!/usr/local/bin/python

"""
This is an example of policy evaluation for a random walk policy.

Example 6.2: Random Walk from the book:
"Reinforcement Learning: An Introduction, Surto and Barto"

The policy is evaluated by dynamic programing and TD(0).

In this example, the policy can start in five states 1, 2, 3, 4, 5 and end in
two states 0 and 6. The allowed transitions between the states are as follwes:

0 <-> 1 <-> 2 <-> 3 <-> 4 <-> 5 <-> 6

The reward for ending in the state 6 is 1.
The reward for ending in the state 0 is 0.

In any state, except the final states, you can take two actions: 'left' and 'right'.
In the final states the policy and episodes end.

Because this example implements the random walk policy then both actions can be
taken with the probability 0.5 .
"""

import random

# Instead of letters, I use number for the states. The states 0 and 6 are the final states.
# The states 1,...,5 are the states A,...,E

states = range(0,7)
finalStates = [0,6]
reward = [ 1 if s == 6 else 0 for s in states]

def policy(s):
"""Random walk policy: ations are 'go left' nand 'go right'."""

return random.choice(['left', 'right'])

def execute_policy(s, a):
"""Change state based on the taken action."""

if a == 'left':
return s - 1
else:
return s + 1

def TD_0(V_star, alpha, gamma, numOfEpisodes = 10000):
"""Use Temporal-Difference Learning to learn V^*."""

for episode in range(numOfEpisodes):
# select random start state
s = random.randint(min(states)+1, max(states)-1)

endOfEpisode = False
while not endOfEpisode:
if s in finalStates:
# evaluate the value of the final state
V_star[s] = V_star[s] + alpha*(reward[s] - V_star[s])

# because we are in the final state then end the episode
endOfEpisode = True
continue
else:
# get an action for this state from the policy
a = policy(s)

# execute policy => take an action
s_prime = execute_policy(s, a)

# evaluate the action
V_star[s] = V_star[s] + alpha*(reward[s] + gamma*V_star[s_prime] - V_star[s])

s = s_prime

def V(s, d = 0):
"""Value function computed by dynamic programing."""

if d > 20:
return 0

if s in finalStates:
return reward[s]

return 0.5*(V(s-1, d+1) + V(s+1, d+1))

###############################################################################
##
## Experiments
##
###############################################################################

gamma = 1

print("TD(0): alpha = 0.15")

# init the value function
V_star = [0.5 for s in states]
TD_0(V_star, 0.15, gamma, 100000)
for s, V_s in enumerate(V_star):
V_s_star = s/6.0
print("V(%d) = %0.3f err = %.3f" % (s, V_s, abs(V_s_star - V_s)))

print("TD(0): alpha = 0.05")

# init the value function
V_star = [0.5 for s in states]
TD_0(V_star, 0.05, gamma, 100000)
for s, V_s in enumerate(V_star):
V_s_star = s/6.0
print("V(%d) = %0.3f err = %.3f" % (s, V_s, abs(V_s_star - V_s)))

print "Dynamic programing:"
for s in states:
V_s_star = s/6.0
V_s = V(s)
print("V(%d) = %0.3f err = %.3f" % (s, V_s, abs(V_s_star - V_s)))

Monday, March 30, 2009

Loebner Prize at InterSpeech 2009

I have recetly written an article about the Loebner Prize contest. The contest has been organized every year since 1991 and it will be held in conjunction with InterSpeech, in Brighton, UK this year. Check out the whole article!

Tuesday, March 24, 2009

Difference between the Standard Deviation, Standard Error and Confidence Itervals

In its simplest:
  • the standard deviation represents the variability of input values,
  • the standard error (of the mean) represents variability of computed mean,
  • the confidence intervals represents where the 'true' mean value might lie.
Computation:
  • the standard deviation is computed from the variance of your data - input values,
  • the standard error is computed from the standard deviation,
  • the confidence intervals are computed from the standard error.
Read this:

Many people confuse the standard deviation (SD) and the standard error of the mean (SE) and are unsure which, if either, to use in presenting data in graphical or tabular form. The SD is an index of the variability of the original data points and should be reported in all studies. The SE reflects the variability of the mean values, as if the study were repeated a large number of times. By itself, the SE is not particularly useful; however, it is used in constructing 95% and 99% confidence intervals (CIs), which indicate a range of values within which the “true” value lies. The CI shows the reader how accurate the estimates of the population values actually are. If graphs are used, error bars equal to plus and minus 2 SEs (which show the 95% CI) should be drawn around mean values. Both statistical significance testing and CIs are useful because they assist the reader in determining the meaning of the findings.

(Can J Psychiatry 1996;41:498–502)

Monday, March 16, 2009

Google Testing Blog: TotT: EXPECT vs. ASSERT

Google commented on different types of assertations in their testing framework (Google Testing Blog: EXPECT vs. ASSERT) 

I have found assertations in my code very useful on many occasions; however, I do not see any need for the EXPECT function. If your code is broken then it is broken and there is no point in continuing and testing conditions which are not likely to be met. 

It is like with C++ compiler errors. The most important error is the firtst error. The rest of the erorrs is usually rubish and useless. 

Wednesday, March 11, 2009

Linked list sorted in an ascending order

Again to practice some of skills I wrote linked list sorted in an ascending order. It is very simple example and it performs only two operation:
  • inserting an item into the appropriate position in the list
  • printing the whole list on the screen
The following code is placed in the public domain. Use it as you wish. The code can be downloaded with the solution project file for MS Visual Studio 2008 from here.

// LinkedList.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include

/*
The supplied C code fragment is intended to implement a linked list of
integers stored in ascending order. Each element of the list is a struct
of type Item holding the integer value, a pointer pred to the previous
element, if any, and a pointer succ to the succeeding element, if any. The
variable head points to the ?rst element in the list, and tail points to
the last element. Initially, both head and tail are NULL. The function
insert is intended to insert its argument x into the list. If x is already in
the list, insert should do nothing.
*/

using namespace std;

struct Item {
int value; // the integer value
Item* succ; // succeeding value
Item* pred; // preceding value
};

Item* head = NULL;
Item* tail = NULL;

Item* createItem(int x, Item* succ, Item* pred)
{
Item* item = new Item;
item->value = x;
item->succ = succ;
item->pred = pred;

return item;
}

/**
Insert x into linked list

There are 5 possible cases, any four of these will suffice
- list is initially empty
- x is smaller than any number in the list
- x is larger than any number in the list
- x is already in the list
- the normal case i.e. none of the above

*/

void insert(int x)
{
Item *p = head , *q;
if(!p) {
// add the first item
head = tail = createItem(x,NULL,NULL);
}
else if (x > tail->value){
// add the largest item in the list
tail->succ = createItem(x,NULL,tail);
tail = tail->succ;
}
else{
while (x > p->value)
p = p->succ;

if (x==p->value)
// do not add itme because it already exists
return;

// add the item before p because x < q =" createItem(x,">pred); // create item, pass the succesor and the predcesor
q->succ->pred = q; // modify the succesor of q so that it points to q
q->pred->succ = q; // modify the predcesor of q so that it points to q
}
}

void printList(void)
{
Item* p = head;
int counter = 0;

cout << "------------------------------------------------------------" <<>value << p =" p-">succ;
}
}

int _tmain(int argc, _TCHAR* argv[])
{
cout << "Ascending linked list demonstration." << i =" 0;">


The output of the program is as follows:

Ascending linked list demonstration.
--------------------------------------
Item number:0 Item value:1
--------------------------------------
Item number:0 Item value:1
Item number:1 Item value:10
--------------------------------------
Item number:0 Item value:1
Item number:1 Item value:5
Item number:2 Item value:10
--------------------------------------
Item number:0 Item value:1
Item number:1 Item value:5
Item number:2 Item value:10


Wednesday, January 21, 2009

Viterbi Algorithm in C++ and using STL

To practice my C++ and STL skills, I implemented the Viterbi algorithm example from the Wikipedia page: http://en.wikipedia.org/wiki/Viterbi_algorithm. The original algorithm was implemented in Python. I reimplemented the example in C++ and I used STL (mainly vector and map classes).  This code is in public-domain. So, use it as you want. 

The complete solution for MS Visual C++ 2008 can be found at http://filip.jurcicek.googlepages.com/ViterbiSTL.rar

// ViterbiSTL.cpp : is an C++ and STL implementatiton of the Wikipedia example
// Wikipedia: http://en.wikipedia.org/wiki/Viterbi_algorithm#A_concrete_example
// It as accurate implementation as it was possible

#include "stdafx.h"

#include "string"
#include "vector"
#include "map"
#include "iostream"

using namespace std;

//states = ('Rainy', 'Sunny')
// 
//observations = ('walk', 'shop', 'clean')
// 
//start_probability = {'Rainy': 0.6, 'Sunny': 0.4}
// 
//transition_probability = {
//   'Rainy' : {'Rainy': 0.7, 'Sunny': 0.3},
//   'Sunny' : {'Rainy': 0.4, 'Sunny': 0.6},
//   }
// 
//emission_probability = {
//   'Rainy' : {'walk': 0.1, 'shop': 0.4, 'clean': 0.5},
//   'Sunny' : {'walk': 0.6, 'shop': 0.3, 'clean': 0.1},
//   }

vector states;
vector observations;
map start_probability;
map> transition_probability;
map> emission_probability;

class Tracking {
public:
  double prob;
  vector v_path;
  double v_prob;

  Tracking() {
    prob = 0.0;
    v_prob = 0.0;
  }

  Tracking(double p, vector & v_pth, double v_p) {
    prob = p;
    v_path = v_pth;
    v_prob = v_p;
  }
};

//////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////

void init_variables(void) {
  states.push_back("Rainy");
states.push_back("Sunny");

  observations.push_back("walk");
  observations.push_back("shop");
  observations.push_back("clean");

  start_probability["Rainy"] = 0.6;
  start_probability["Sunny"] = 0.4;

  transition_probability["Rainy"]["Rainy"] = 0.7;
  transition_probability["Rainy"]["Sunny"] = 0.3;
  transition_probability["Sunny"]["Rainy"] = 0.4;
  transition_probability["Sunny"]["Sunny"] = 0.6;

  emission_probability["Rainy"]["walk"] = 0.1;
  emission_probability["Rainy"]["shop"] = 0.4;
  emission_probability["Rainy"]["clean"] = 0.5;
  emission_probability["Sunny"]["walk"] = 0.6;
  emission_probability["Sunny"]["shop"] = 0.3;
  emission_probability["Sunny"]["clean"] = 0.1;
}

//////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////

void print_variables(void) {
  // print states
  cout << "States:" <<>
  for(vector::iterator i=states.begin();i!=states.end();i++) {
    cout << "S: " << (*i) <<>
  }
  // print observations
  cout << "Observations:" <<>
  for(vector::iterator i=observations.begin();i!=observations.end();i++) {
    cout << "O: " << (*i) <<>
  }

  // print start probabilities
  cout << "Start probabilities:" <<>
  for(map::iterator i=start_probability.begin();i!=start_probability.end();i++) {
    cout << "S: " << (*i).first << " P: " << (*i).second <<>
  }

  // print transition_probability
  cout << "Transition probabilities:" <<>
  for(map>::iterator i=transition_probability.begin();i!=transition_probability.end();i++) {
    for(map::iterator j=(*i).second.begin();j!=(*i).second.end();j++) {
      cout << "FS: " << (*i).first << " TS: " << (*j).first << " P: " << (*j).second <<>
    }
  }

  // print emission probabilities
  cout << "Emission probabilities:" <<>
  for(int i=0; i
    for(int j=0; j
      cout << "FS: " <<>
        " P: " <<>
    }
  }
}

//this method compute total probability for observation, most likely viterbi path 
//and probability of such path
void forward_viterbi(vector obs, vector states, map start_p, 
                     map> trans_p, 
                     map> emit_p) {
  map T;

  for(vector::iterator state=states.begin(); state!=states.end();state++) {
    vector v_pth;
    v_pth.push_back(*state);

    T[*state] = Tracking(start_p[*state], v_pth, start_p[*state]);
  }

  for(vector::iterator output=obs.begin(); output!=obs.end();output++) {
    map U;

    for(vector::iterator next_state=states.begin(); next_state!=states.end(); next_state++) {
      Tracking next_tracker;

      for(vector::iterator source_state=states.begin(); source_state!=states.end(); source_state++) {
        Tracking source_tracker = T[*source_state];

        double p = emit_p[*source_state][*output]*trans_p[*source_state][*next_state];
        source_tracker.prob *= p;
        source_tracker.v_prob *= p;

        next_tracker.prob += source_tracker.prob;

        if(source_tracker.v_prob > next_tracker.v_prob) {
          next_tracker.v_path = source_tracker.v_path;
          next_tracker.v_path.push_back(*next_state);
          next_tracker.v_prob = source_tracker.v_prob;
        }
      }

      U[*next_state] = next_tracker;
    }

    T = U;
  }

  // apply sum/max to the final states
  Tracking final_tracker;

  for(vector::iterator state=states.begin(); state!=states.end(); state++) {
    Tracking tracker = T[*state];

    final_tracker.prob += tracker.prob;

    if(tracker.v_prob > final_tracker.v_prob) {
      final_tracker.v_path = tracker.v_path;
      final_tracker.v_prob = tracker.v_prob;
    }
  }

  cout << "Total probability of the observation sequence: " <<>
  cout << "Probability of the Viterbi path: " <<>
  cout << "The Viterbi path: " <<>
  for(vector::iterator state=final_tracker.v_path.begin(); state!=final_tracker.v_path.end(); state++) {
    cout << "VState: " << *state <<>
  }
}

int _tmain(int argc, _TCHAR* argv[])
{
  cout << "Viterbi STL example" <<>
  
  init_variables();
  print_variables();

  forward_viterbi(observations, 
                  states, 
                  start_probability, 
                  transition_probability, 
                  emission_probability);

cout << "End" <<>

  string end;
  cin >> end;

  return 0;
}