Brian Tanner :: brian@tannerpages.com
This document describes how to use the C/C++ RL-Glue Codec, a software library that provides socket-compatibility with the RL-Glue Reinforcement Learning software library.
For general information and motivation about the RL-Glue1 project, please refer to the documentation provided with that project.
This codec will allow you to create agents, environments, and experiment programs in C and/or C++.
This software project is licensed under the Apache-2.02 license. We're not lawyers, but our intention is that this code should be used however it is useful. We'd appreciate to hear what you're using it for, and to get credit if appropriate.
This project has a home here:
http://glue.rl-community.org/Home/Extensions/c-c-codec
This document should be used in combinations with the technical manual for RL-Glue 3.0. The RL-Glue 3.0 technical manual explains the data types, function prototypes, etc. for the main C/C++ RL-Glue implementation that is used
by this codec. The RL-Glue 3.0 technical manual can be found in the docs section of the RL-Glue project or online:
http://rl-glue.googlecode.com/svn/trunk/docs/tech_html/index.html
All of the official downloads of the C/C++ codec can be found here:
http://code.google.com/p/rl-glue-ext/wiki/CandCPP
You may also check the code out directly from the subversion:
svn checkout http://rl-glue-ext.googlecode.com/svn/trunk/projects/codecs/C C-Codec
The steps are:
>$ ./configure >$ make >$ sudo make install
Provided everything goes well, the headers have now been installed to /usr/local/include and the libs to /usr/local/lib.
NOTE: On many Linux systems, /usr/local is not actually on the library and header search paths by default, but /usr surely is. In this case, you may want to follow the instructions in Section 1.4.3 system with -prefix=/usr.
If configure can't find RL-Glue installed on your machine, it will give you an error like the following:
checking for rlConnect in -lrlgluenetdev... no configure: error: RL-Glue library not found. You must have RL-Glue installed to use this codec. If you have not downloaded it please see http://glue.rl-community.org/ If you do have it installed in a non-standard location you may need to use the --with-rl-glue=/path/to/rlglue command line switch to specify where the rl-glue root is located.
If you installed RL-Glue to some place other than /usr/local, say /Users/btanner/tmp/rlglue, you could do:
>$ ./configure --with-rl-glue=/Users/btanner/tmp/rlglue >$ make >$ sudo make install
If you don't have sudo or root access on the target machine, you can install the codec in your home directory (or other directory you have access to). If you install to a custom location, you will need set your CFLAGS and LDFLAGS variables appropriately when compiling your agents, environments, and experiments. See Section 2.2 for more information.
For example, maybe we want to install the codec to /Users/btanner/tmp/rlglue. This will not clobber RL-Glue if it is already installed to this location, it will install beside it. The commands are:
>$ ./configure --prefix=/Users/btanner/tmp/rlglue >$ make >$ make install
Provided everything goes well, the headers and libraries have been respectively installed to
/Users/btanner/tmp/rlglue/include and /Users/btanner/tmp/rlglue/lib.
>$ ./configure >$ sudo make uninstall
This will remove all of the headers and libraries from /usr/local.
>$ ./configure --prefix=/Users/btanner/tmp/rlglue >$ make uninstall
That's it! This will remove all of the headers and libraries from /Users/btanner/tmp/rlglue.
The skeleton-example contains all of the bare-bones plumbing that is required to create an agent/environment/experiment with this codec and might be a good starting point for creating your own components.
The mines-sarsa-sample contains a fully functional tabular Sarsa learning algorithm, a discrete-observation grid world problem, and an experiment program that can run these together and gather results. More details below in Section 3.1.
the custom_integrated_env in Section 3.1.4 shows how you can adapt this codec to help programs not originally meant for RL-Glue (Atari emulators, real-time strategy games, etc.) work with RL-Glue.
In the following sections, we will describe the skeleton project. Running and using the mines-sarsa-sample is analogous.
The README file in the specific examples directories may also be helpful in getting started.
The pertinent files are:
examples/skeleton-example/SkeletonAgent.c examples/skeleton-example/Makefile
This agent does not learn anything and randomly chooses integer action 0 or 1.
If RL-Glue and this codec have been installed in the default location, /usr/local, then you can compile and run the agent like:
>$ cd examples/skeleton-example >$ make >$ ./SkeletonAgent
You will see something like:
RL-Glue C Agent Codec Version 1.0-alpha-3, Build 192:208M Connecting to host=127.0.0.1 on port=4096...
This means that the SkeletonAgent is running, and trying to connect to the rl_glue executable server on the local machine through port 4096!
You can kill the process by pressing CTRL-C on your keyboard.
See Section 2.2 if RL-Glue or this Codec are not installed in default locations.
The Skeleton agent is very simple and well documented, so we won't spend any more time talking about it in these instructions. Please open it up and take a look.
We will spend a little bit talking about how to compile the agent, because not everyone is comfortable with using a Makefile. To compile the agent from the command line, you could do:
>$ cc SkeletonAgent.c -lrlutils -lrlagent -o SkeletonAgent
On some platforms, you may need to add -lrlgluenetdev
It might be useful to break this down a little bit:
To do it on the command line:
>$ CFLAGS=-I/Users/joe/glue/include LDFLAGS=-L/Users/joe/glue/lib make
That might turn out to be quite a hassle while you are developing. In that case, you can either update the Makefile to include these flags, or set an environment variable. If you are using the bash shell you can export the environment variables:
>$ export CFLAGS=-I/Users/joe/glue/include >$ export LDFLAGS=-L/Users/joe/glue/lib >$ make
In some cases, you may be able to compile and link your programs without incident, but you receive shared library loading errors when you try to execute them, as mentioned in Gotchas! (Section 2.5.2).
The reason for these errors is that the search path of the loader does not include /usr/local/lib. This problem has both temporary and permanent fixes.
To temporarily fix the problem, you can set LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (OS X) environment variables, like:
>$ export LD_LIBRARY_PATH=/Users/joe/glue/lib
In some cases (64-bit linux) you may have to use this approach even when RL-Glue and this codec are installed in the default locations:
>$ export LD_LIBRARY_PATH=/usr/local/lib
When you open a new terminal window, all of these environment variables will be lost unless you put the appropriate export lines in your shell startup script.
The permanent solution requires root or sudo access to the machine. You can edit the file /etc/ld.so.conf and add a line like: /usr/local/lib to the file. Then, if you call sudo ldconfig, the loader will know to look there for libraries in the future.
The pertinent files are:
examples/skeleton-example/SkeletonEnvironment.c examples/skeleton-example/Makefile
This environment is episodic, with 21 states, labeled
. States {0, 20} are terminal and return rewards of {-1, +1} respectively. The other states return reward of 0.
There are two actions, {0, 1}. Action 0 decrements the state number, and action 1 increments it. The environment starts in state 10.
If RL-Glue and this codec have been installed in the default location, /usr/local, then you can compile and run the environment like:
>$ cd examples/skeleton-example >$ make >$ ./SkeletonEnvironment
You will see something like:
RL-Glue C Environment Codec Version 1.0-alpha-3, Build 192:208M Connecting to host=127.0.0.1 on port=4096...
This means that the SkeletonEnvironment is running, and trying to connect to the rl_glue executable server on the local machine through port 4096!
You can kill the process by pressing CTRL-C on your keyboard.
See Section 2.2 if RL-Glue or this Codec are not installed in default locations.
The Skeleton environment is very simple and well documented, so we won't spend any more time talking about it in these instructions. Please open it up and take a look.
Compiling the environment is almost identical to compiling the skeleton agent, except you need to link to the RLEnvironment library instead of RLAgent.
>$ cc SkeletonEnvironment.c -lrlutils -lrlenvironment -o SkeletonEnvironment
On some platforms, you may need to add -lrlgluenetdev
The pertinent files are:
examples/skeleton-example/SkeletonExperiment.c examples/skeleton-example/Makefile
This experiment runs RL_Episode a few times, sends some messages to the agent and environment, and then steps through one episode using RL_step.
If RL-Glue and this codec have been installed in the default location, /usr/local, then you can compile and run the experiment like:
>$ cd examples/skeleton-example >$ make >$ ./SkeletonExperiment
You will see something like:
RL-Glue C Experiment Codec Version 1.0-alpha-3, Build 192:208M Connecting to host=127.0.0.1 on port=4096...
This means that the SkeletonExperiment is running, and trying to connect to the rl_glue executable server on the local machine through port 4096!
You can kill the process by pressing CTRL-C on your keyboard.
See Section 2.2 if RL-Glue or this Codec are not installed in default locations.
The Skeleton experiment is very simple and well documented, so we won't spend any more time talking about it in these instructions. Please open it up and take a look.
Compiling the experiment is almost identical to compiling the skeleton agent, except you need to link to the RLExperiment library instead of RLAgent.
>$ cc SkeletonExperiment.c -lrlutils -lrlexperiment -o SkeletonExperiment
On some platforms, you may need to add -lrlgluenetdev
>$ ./SkeletonAgent: error while loading shared libraries: librlagent-1:0:0.so.1: cannot open shared object file: No such file or directory
So far this has only happened on 64-bit Linux, both a machine at the University and a personal install of 64-bit Ubuntu, Hardy Heron.
The reason for the error is that /usr/local/lib is not in the the loader's search path. If this happens, the operating system might have an alternate search path, and might not be looking in /usr/local/lib for libraries. You can troubleshoot this problem by doing:
>$ LD_DEBUG=libs ./RL_agentIf you see that /usr/local/lib is not in the search path, you may want to add it to your library search path using LDFLAGS or LD_LIBRARY_PATH. See Section 2.2 for more information.
You can either execute the following four commands in the same terminal window if you put each on in the background (& on the end), or run them in separate terminal windows:
>$ rl_glue >$ skeleton-example/SkeletonAgent >$ skeleton-example/SkeletonEnvironment >$ skeleton-example/SkeletonExperiment
If RL-Glue is not installed in the default location, you'll have to start the rl_glue executable server using its full path (unless it's in your PATH environment variable):
>$ /path/to/rl-glue/bin/rl_glue &
You should see output like the following if it worked:
>$ rl_glue & RL-Glue Version 3.0-alpha-3, Build 848:852M RL-Glue is listening for connections on port=4096 >$ skeleton-example/SkeletonAgent & RL-Glue C Agent Codec Version 1.0-alpha-3, Build 192:208M Connecting to host=127.0.0.1 on port=4096... RL-Glue C Agent Codec :: Connected RL-Glue :: Agent connected. >$ skeleton-example/SkeletonEnvironment & RL-Glue C Environment Codec Version 1.0-alpha-3, Build 192:208M Connecting to host=127.0.0.1 on port=4096... RL-Glue C Environment Codec :: Connected RL-Glue :: Environment connected. $> skeleton-example/SkeletonExperiment Experiment starting up! RL-Glue C Experiment Codec Version 1.0-alpha-3, Build 192:208M Connecting to host=127.0.0.1 on port=4096... RL-Glue C Experiment Codec :: Connected RL-Glue :: Experiment connected. RL_init called, the environment sent task spec: VERSION RL-Glue-3.0 PROBLEMTYPE episodic DISCOUNTFACTOR 1.0 OBSERVATIONS INTS (0 20) ACTIONS INTS (0 1) REWARDS (-1.0 1.0) EXTRA skeleton_environment(C/C++) by Brian Tanner. ----------Sending some sample messages---------- Agent responded to "what is your name?" with: my name is skeleton_agent! Agent responded to "If at first you don't succeed; call it version 1.0" with: I don't know how to respond to your message Environment responded to "what is your name?" with: my name is skeleton_environment! Environment responded to "If at first you don't succeed; call it version 1.0" with: I don't know how to respond to your message ----------Running a few episodes---------- Episode 0 100 steps 0.000000 total reward 0 natural end Episode 1 44 steps -1.000000 total reward 1 natural end Episode 2 18 steps -1.000000 total reward 1 natural end Episode 3 100 steps 0.000000 total reward 0 natural end Episode 4 50 steps 1.000000 total reward 1 natural end Episode 5 1 steps 0.000000 total reward 0 natural end Episode 6 28 steps 1.000000 total reward 1 natural end ----------Stepping through an episode---------- First observation and action were: 10 1 ----------Summary---------- It ran for 144 steps, total reward was: -1.000000
More details about the mines-sarsa sample project can be found at their RL-Library home:
http://library.rl-community.org/packages/mines-sarsa-sample
The task specification string5 is manually created because there is not yet a task spec builder for C/C++.
The SARSA agent parses the task specification string using the C/C++ task spec parser. This agent can receive special messages from the experiment program to pause/unpause learning, pause/unpause exploring, save the current value function to a file, and load the the value function from a file.
The sample experiment then tells the agent to save the value function to a file, and then resets the experiment (and agent) to initial conditions. After verifying that the agent's initial policy is bad, the experiment tells the agent to load the value function from the file. The agent is evaluated again using this previously-learned value function, and performance is dramatically better.
Finally, the experiment sends a message to specify that the environment should use a fixed (instead of random) starting state, and runs the agent from that fixed start state for a while.
Two such examples that have happened already are Marc Lanctot's real time strategy engine that was used in the 2008 RL-Competition, and also the project to connect the Stella Atari emulator to RL-Glue.
We support a path for integrating these projects with RL-Glue that involves customizing this C/C++ codec, rather than trying too hard to link to it. We have provided an example in: examples/custom_integrated_env.
The relevant files are:
TheGame.h
TheGame.c
Custom_Integrated_Env_Codec.c
The idea in the example is that you have a game (called TheGame) which was never meant to be used with RL-Glue. To integrate it, a copy of src/RL_client_environment.c was modified to work specifically with TheGame. For example, all of the calls to env_init, env_start, env_step, etc. were replaced with calls to functions defined in TheGame. The input and output to those TheGame functions need to be wrapped up a little bit, but the end result is a fairly easy way to connect TheGame to RL-Glue!
Check out the source code in the example, and feel free to ask questions in the RL-Glue Google Group!
The main thing to realize is that agents, environments, and experiment programs are completely interchangeable between direct-compile (RL-Glue project) and socket-communication (this codec). An agent written for either one will be compiled the same way, the only difference is what library it is linked to. So, 95% of the technical manual from the RL-Glue project applies perfectly to people using this codec!
In these cases, you can tell this codec to connect to custom port/host combinations using the environment variables RLGLUE_HOST and RLGLUE_PORT. These are both optional parameters.
For example, try the following code:
> $ RLGLUE_HOST=69.64.159.1 RLGLUE_PORT=1025 skeleton-example/SkeletonAgent
That command could give outcome:
RL-Glue C Agent Codec Version 2.0-RC1, Build 277 Connecting to host=69.64.159.1 on port=1025...
Alternatively, you can actually use a host name:
> $ RLGLUE_HOST=rlai.net skeleton-example/SkeletonAgent
That command could give outcome:
RL-Glue C Agent Codec Version 2.0-RC1, Build 277 Connecting to host=69.64.159.1 on port=4096...
If you don't like typing them every time, you can export them:
> $ export RLGLUE_HOST=rlai.net > $ export RLGLUE_PORT=1025 > $ skeleton-example/SkeletonAgent
The online FAQ may be more current than this document, which may have been distributed some time ago.
We're happy to answer any questions about RL-Glue. Of course, try to search through previous messages first in case your question has been answered before.
Brian Tanner has since grabbed the torch and has continued to develop the codec.
Special thanks to Scott Livingston for creating the new C/C++ task spec parser.
Revision Number: $Rev: 681 $ Last Updated By: $Author: brian@tannerpages.com $ Last Updated : $Date: 2009-02-08 20:03:55 -0700 (Sun, 08 Feb 2009) $ $URL: https://rl-glue-ext.googlecode.com/svn/trunk/projects/codecs/C/docs/C-Codec.tex $
This document was generated using the LaTeX2HTML translator Version 2002-2-1 (1.71)
Copyright © 1993, 1994, 1995, 1996,
Nikos Drakos,
Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999,
Ross Moore,
Mathematics Department, Macquarie University, Sydney.
The command line arguments were:
latex2html C-Codec.tex -split 0 -dir html -mkdir -title 'RL-Glue C/C++ Codec' -local_icons -math
The translation was initiated by Brian Tanner on 2009-02-09