This web page provides selections of code used for experiments in my dissertation. This code is not well documented or elegant, but some may prefer it to reading the psudo-code in text. Feel free to use or build upon anything here, but please do cite the dissertation (or the original paper) if you find it useful.

Novel Domains

All domains are detailed in Chapter 3.
  • 3D Mountain Car

    These two files describe the environment (domain) which can be used in the RL-Glue framework: MountainCar3DSym.cc MountainCar3DSym.h
  • Keepaway

    Keepaway can be downloaded from here, along with tutorials for instaliation and programming Keepaway agents.
    • Inaccurate 3 vs. 2 and Inaccurate 4 vs. 3

      To change the default 3 vs. 2 or 4 vs. 3 games so that Keepers have inaccurate passing actuators, search for the line
      ACT->putCommandInQueue( soc = directPass( tmPos, PASS_NORMAL ) );
      in KeepawayPlayer.cc, and replace it with the line
      ACT->putCommandInQueue( soc = directPass( tmPos, PASS_FAST ) );
    • 3 vs. 2 XOR Keepaway

      To change the default 3 vs. 2 game to XOR Keepaway, KeepawayPlayer.cc should be replaced with this file and the changes described here in should be made to WorldModelUpdate.cc and WorldModel.h.
  • Ringworld

    A Java implementation of the environment can be found here.
  • Knight Joust

    The Knight Joust Java implementation is here.

Selected Algorithms

  • Value function transfer: CMAC transfer functional for 3 vs. 2 Keepaway

    Relies on Version 2.0 of the Tile Coding Software as the CMAC implementation. These two functions should be added to CMAC.cc (and CMAC.h) and then called after CMAC/RBF weights are loaded into a target task player from a source task player, but before learing in the target task has started.
  • Policy Transfer for NEAT (neural network action selectors)

    After training a source population of neural networks, the network structure needs to be augmented for the target task, and weights need to be set. This program was used to change a neural network (called a genome by NEAT) that was trained in the 2-job-type Server Job Scheduling task so that it is appropriate for the 4-job-type Server Job Scheduling task. The program is called once and processes the population of neural networks, as output by NEAT. UpdatePopulation.cc