Reinforcement learning in non-markovian environments using automatic discovery of subgoals

Le Tien Dung; Takashi Komeda; Motoki Takagi

doi:10.1109/SICE.2007.4421430

Reinforcement learning in non-markovian environments using automatic discovery of subgoals

Le Tien Dung, Takashi Komeda, Motoki Takagi

Department of Bioscience and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

6 Citations (Scopus)

Abstract

Learning time is always a critical issue in Reinforcement Learning, especially when Recurrent Neural Networks (RNNs) are used to predict Q values. By creating useful subgoals, we can speed up learning performance. In this paper, we propose a method to accelerate learning in non-Markovian environments using automatic discovery of subgoals. Once subgoals are created, sub-policies use RNNs to attain them. Then learned RNNs are integrated into the main RNN as experts. Finally, the agent continues to learn using its new policy. Experiment results of the E maze problem and the virtual office problem show the potential of this approach.

Original language	English
Title of host publication	SICE Annual Conference, SICE 2007
Pages	2601-2605
Number of pages	5
DOIs	https://doi.org/10.1109/SICE.2007.4421430
Publication status	Published - 2007 Dec 1
Event	SICE(Society of Instrument and Control Engineers)Annual Conference, SICE 2007 - Takamatsu, Japan Duration: 2007 Sept 17 → 2007 Sept 20

Publication series

Name	Proceedings of the SICE Annual Conference

Conference

Conference	SICE(Society of Instrument and Control Engineers)Annual Conference, SICE 2007
Country/Territory	Japan
City	Takamatsu
Period	07/9/17 → 07/9/20

Keywords

Selected keywords relevant to the subject

ASJC Scopus subject areas

Control and Systems Engineering
Computer Science Applications
Electrical and Electronic Engineering

Access to Document

10.1109/SICE.2007.4421430

Cite this

Dung, LT, Komeda, T & Takagi, M 2007, Reinforcement learning in non-markovian environments using automatic discovery of subgoals. in SICE Annual Conference, SICE 2007., 4421430, Proceedings of the SICE Annual Conference, pp. 2601-2605, SICE(Society of Instrument and Control Engineers)Annual Conference, SICE 2007, Takamatsu, Japan, 07/9/17. https://doi.org/10.1109/SICE.2007.4421430

@inproceedings{bf051816effb4203ae6c9457ff93b642,

title = "Reinforcement learning in non-markovian environments using automatic discovery of subgoals",

abstract = "Learning time is always a critical issue in Reinforcement Learning, especially when Recurrent Neural Networks (RNNs) are used to predict Q values. By creating useful subgoals, we can speed up learning performance. In this paper, we propose a method to accelerate learning in non-Markovian environments using automatic discovery of subgoals. Once subgoals are created, sub-policies use RNNs to attain them. Then learned RNNs are integrated into the main RNN as experts. Finally, the agent continues to learn using its new policy. Experiment results of the E maze problem and the virtual office problem show the potential of this approach.",

keywords = "Selected keywords relevant to the subject",

author = "Dung, {Le Tien} and Takashi Komeda and Motoki Takagi",

year = "2007",

month = dec,

day = "1",

doi = "10.1109/SICE.2007.4421430",

language = "English",

isbn = "4907764286",

series = "Proceedings of the SICE Annual Conference",

pages = "2601--2605",

booktitle = "SICE Annual Conference, SICE 2007",

note = "SICE(Society of Instrument and Control Engineers)Annual Conference, SICE 2007 ; Conference date: 17-09-2007 Through 20-09-2007",

}

TY - GEN

T1 - Reinforcement learning in non-markovian environments using automatic discovery of subgoals

AU - Dung, Le Tien

AU - Komeda, Takashi

AU - Takagi, Motoki

PY - 2007/12/1

Y1 - 2007/12/1

N2 - Learning time is always a critical issue in Reinforcement Learning, especially when Recurrent Neural Networks (RNNs) are used to predict Q values. By creating useful subgoals, we can speed up learning performance. In this paper, we propose a method to accelerate learning in non-Markovian environments using automatic discovery of subgoals. Once subgoals are created, sub-policies use RNNs to attain them. Then learned RNNs are integrated into the main RNN as experts. Finally, the agent continues to learn using its new policy. Experiment results of the E maze problem and the virtual office problem show the potential of this approach.

AB - Learning time is always a critical issue in Reinforcement Learning, especially when Recurrent Neural Networks (RNNs) are used to predict Q values. By creating useful subgoals, we can speed up learning performance. In this paper, we propose a method to accelerate learning in non-Markovian environments using automatic discovery of subgoals. Once subgoals are created, sub-policies use RNNs to attain them. Then learned RNNs are integrated into the main RNN as experts. Finally, the agent continues to learn using its new policy. Experiment results of the E maze problem and the virtual office problem show the potential of this approach.

KW - Selected keywords relevant to the subject

UR - http://www.scopus.com/inward/record.url?scp=50249160640&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=50249160640&partnerID=8YFLogxK

U2 - 10.1109/SICE.2007.4421430

DO - 10.1109/SICE.2007.4421430

M3 - Conference contribution

AN - SCOPUS:50249160640

SN - 4907764286

SN - 9784907764289

T3 - Proceedings of the SICE Annual Conference

SP - 2601

EP - 2605

BT - SICE Annual Conference, SICE 2007

T2 - SICE(Society of Instrument and Control Engineers)Annual Conference, SICE 2007

Y2 - 17 September 2007 through 20 September 2007

ER -

Reinforcement learning in non-markovian environments using automatic discovery of subgoals

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this