Addressing the local minima problem by output monitoring and modification algorithms

Sin Chun Ng; Chi Chung Cheung; Andrew Kwok Fai Lui; Hau Ting Tse

doi:10.1007/978-3-642-31346-2_24

Addressing the local minima problem by output monitoring and modification algorithms

Sin Chun Ng, Chi Chung Cheung, Andrew Kwok Fai Lui, Hau Ting Tse

Hong Kong Metropolitan University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

Abstract

This paper proposes a new approach called output monitoring and modification (OMM) to address the local minimum problem for existing gradient-descent algorithms (like BP, Rprop and Quickprop) in training feed-forward neural networks. OMM monitors the learning process. When the learning process is trapped into a local minimum, OMM changes some incorrect output values to escape from such local minimum. This modification can be repeated with different parameter settings until the learning process converges to the global optimum. The simulation experiments show that a gradient-descent learning algorithm with OMM has a much better global convergence capability than those without OMM but their convergence rates are similar. In one benchmark problem (application), the global convergence capability was increased from 1% to 100%.

Original language	English
Title of host publication	Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings
Pages	206-216
Number of pages	11
Edition	PART 1
DOIs	https://doi.org/10.1007/978-3-642-31346-2_24
Publication status	Published - 2012
Event	9th International Symposium on Neural Networks, ISNN 2012 - Shenyang, China Duration: 11 Jul 2012 → 14 Jul 2012

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 1
Volume	7367 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	9th International Symposium on Neural Networks, ISNN 2012
Country/Territory	China
City	Shenyang
Period	11/07/12 → 14/07/12

Keywords

Quickprop
Rprop
back-propagation
local minimum problem

Access to Document

10.1007/978-3-642-31346-2_24

Cite this

Ng, S. C., Cheung, C. C., Lui, A. K. F., & Tse, H. T. (2012). Addressing the local minima problem by output monitoring and modification algorithms. In Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings (PART 1 ed., pp. 206-216). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7367 LNCS, No. PART 1). https://doi.org/10.1007/978-3-642-31346-2_24

Ng, Sin Chun ; Cheung, Chi Chung ; Lui, Andrew Kwok Fai et al. / Addressing the local minima problem by output monitoring and modification algorithms. Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1. ed. 2012. pp. 206-216 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1).

@inproceedings{37b504805d4346afa5f1a9f13ab3d899,

title = "Addressing the local minima problem by output monitoring and modification algorithms",

abstract = "This paper proposes a new approach called output monitoring and modification (OMM) to address the local minimum problem for existing gradient-descent algorithms (like BP, Rprop and Quickprop) in training feed-forward neural networks. OMM monitors the learning process. When the learning process is trapped into a local minimum, OMM changes some incorrect output values to escape from such local minimum. This modification can be repeated with different parameter settings until the learning process converges to the global optimum. The simulation experiments show that a gradient-descent learning algorithm with OMM has a much better global convergence capability than those without OMM but their convergence rates are similar. In one benchmark problem (application), the global convergence capability was increased from 1% to 100%.",

keywords = "Quickprop, Rprop, back-propagation, local minimum problem",

author = "Ng, {Sin Chun} and Cheung, {Chi Chung} and Lui, {Andrew Kwok Fai} and Tse, {Hau Ting}",

year = "2012",

doi = "10.1007/978-3-642-31346-2_24",

language = "English",

isbn = "9783642313455",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "PART 1",

pages = "206--216",

booktitle = "Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings",

edition = "PART 1",

note = "9th International Symposium on Neural Networks, ISNN 2012 ; Conference date: 11-07-2012 Through 14-07-2012",

}

Ng, SC, Cheung, CC, Lui, AKF & Tse, HT 2012, Addressing the local minima problem by output monitoring and modification algorithms. in Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 1, vol. 7367 LNCS, pp. 206-216, 9th International Symposium on Neural Networks, ISNN 2012, Shenyang, China, 11/07/12. https://doi.org/10.1007/978-3-642-31346-2_24

Addressing the local minima problem by output monitoring and modification algorithms. / Ng, Sin Chun; Cheung, Chi Chung; Lui, Andrew Kwok Fai et al.
Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1. ed. 2012. p. 206-216 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7367 LNCS, No. PART 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Addressing the local minima problem by output monitoring and modification algorithms

AU - Ng, Sin Chun

AU - Cheung, Chi Chung

AU - Lui, Andrew Kwok Fai

AU - Tse, Hau Ting

PY - 2012

Y1 - 2012

N2 - This paper proposes a new approach called output monitoring and modification (OMM) to address the local minimum problem for existing gradient-descent algorithms (like BP, Rprop and Quickprop) in training feed-forward neural networks. OMM monitors the learning process. When the learning process is trapped into a local minimum, OMM changes some incorrect output values to escape from such local minimum. This modification can be repeated with different parameter settings until the learning process converges to the global optimum. The simulation experiments show that a gradient-descent learning algorithm with OMM has a much better global convergence capability than those without OMM but their convergence rates are similar. In one benchmark problem (application), the global convergence capability was increased from 1% to 100%.

AB - This paper proposes a new approach called output monitoring and modification (OMM) to address the local minimum problem for existing gradient-descent algorithms (like BP, Rprop and Quickprop) in training feed-forward neural networks. OMM monitors the learning process. When the learning process is trapped into a local minimum, OMM changes some incorrect output values to escape from such local minimum. This modification can be repeated with different parameter settings until the learning process converges to the global optimum. The simulation experiments show that a gradient-descent learning algorithm with OMM has a much better global convergence capability than those without OMM but their convergence rates are similar. In one benchmark problem (application), the global convergence capability was increased from 1% to 100%.

KW - Quickprop

KW - Rprop

KW - back-propagation

KW - local minimum problem

UR - http://www.scopus.com/inward/record.url?scp=84865111329&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-31346-2_24

DO - 10.1007/978-3-642-31346-2_24

M3 - Conference contribution

AN - SCOPUS:84865111329

SN - 9783642313455

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 206

EP - 216

BT - Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings

T2 - 9th International Symposium on Neural Networks, ISNN 2012

Y2 - 11 July 2012 through 14 July 2012

ER -

Ng SC, Cheung CC, Lui AKF, Tse HT. Addressing the local minima problem by output monitoring and modification algorithms. In Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1 ed. 2012. p. 206-216. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). doi: 10.1007/978-3-642-31346-2_24

Addressing the local minima problem by output monitoring and modification algorithms

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this