David J. Finton. When Do Differences Matter? On-Line Feature Extraction Through Cognitive Economy. arXiv:cs.LG/0404032, 2004.
You may download the full text of the paper (20 pages) from its arXiv abstract page:
For an intelligent agent to be truly autonomous, it must be able to adapt its representation to the requirements of its task as it interacts with the world. Most current approaches to on-line feature extraction are ad hoc; in contrast, this paper presents an algorithm that bases judgments of state compatibility and state-space abstraction on principled criteria derived from the psychological principle of cognitive economy. The algorithm incorporates an active form of Q-learning, and partitions continuous state-spaces by merging and splitting Voronoi regions. The experiments illustrate a new methodology for testing and comparing representations by means of learning curves. Results from the puck-on-a-hill task demonstrate the algorithm's ability to learn effective representations, superior to those produced by some other, well-known, methods.
This paper presents an alternative goal for value prediction, based on the insight that some action-value errors will have no effect on the agent's ability to perform its task. The agent learns faster when it can generalize over "similar" states: states that agree on the preferred action and expectation of reward. Because similar states may differ on the expected values of non-preferred actions, grouping these states may increase the overall prediction error---even though these differences do not impact the agent's performance in the task. In contrast, some states should be considered incompatible because ignoring their differences leads the agent to make bad decisions; such states must not be grouped together. This paper presents principled criteria for deciding when the differences matter and when they may be ignored.
| David J. Finton |
|
April 17, 2004 |
| ||