Advanced search
Publication Cover
Connection Science Volume 35, 2023 - Issue 1
Submit an article Journal homepage
525
Views
0
CrossRef citations to date
0
Altmetric
Research Article

Impartial competitive learning in multi-layered neural networks

Ryotaro Kamimuraa IT Education Center, Tokai University, Kitakaname, Hiratsuka, Kanagawa, Japan;b Kumamoto Drone Technology and Development, Kamimatsuo, Nishi-ku, Kumamoto, JapanCorrespondence[email protected]
[email protected]
Article: 2174079 | Received 01 Nov 2022, Accepted 24 Jan 2023, Published online: 22 Feb 2023

References

  • Arbabzadah, F., Montavon, G., Müller, K. R., & Samek, W. (2016). Identifying individual facial expressions by deconstructing a neural network. In German Conference on Pattern Recognition (pp. 344–354).
  • Arpit, D., Jastrz Kebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischerl, A., Courville, A., Bengio, Y., & Lacoste-Julien, S. (2017). A closer look at memorization in deep networks. In International Conference on Machine Learning (pp. 233–242).
  • Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K. R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One, 10(7), e0130140. https://doi.org/10.1371/journal.pone.0130140
  • Bai, Y., Wang, H., Tao, Z., Li, K., & Fu, Y. (2022). Dual lottery ticket hypothesis. arXiv preprint, arXiv:2203.04248.
  • Banerjee, A., & Ghosh, J. (2004). Frequency-sensitive competitive learning for scalable balanced clustering on high-dimensional hyperspheres. Neural Networks, IEEE Transactions on, 15(3), 702–719. https://doi.org/10.1109/TNN.2004.824416
  • Bao, G., Lin, M., Sang, X., Hou, Y., Liu, Y., & Wu, Y. (2022). Classification of dysphonic voices in Parkinson's disease with semi-Supervised competitive learning algorithm. Biosensors, 12(7), 502. https://doi.org/10.3390/bios12070502
  • Binder, A., Montavon, G., Lapuschkin, S., Müller, K. R., & Samek, W. (2016). Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks (pp. 63–71).
  • Bogdan, M., & Rosenstiel, W. (2001). Detection of cluster in self-organizing maps for controlling a prostheses using nerve signals. In 9th European Symposium on Artificial Neural Networks. ESANN. Proceedings. D-facto, Evere, Belgium (pp. 131–136).
  • Brugger, D., Bogdan, M., & Rosenstiel, W. (2008). Automatic cluster detection in kohonen's SOM. Neural Networks, IEEE Transactions on, 19(3), 442–459. https://doi.org/10.1109/TNN.2007.909556
  • Bucilu, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model compression. In Proceedings of the 12th Acm Sigkdd International Conference on Knowledge Discovery and Data Mining (pp. 535–541).
  • Chen, X., Cheng, Y., Wang, S., Gan, Z., Liu, J., & Wang, Z. (2021). The elastic lottery ticket hypothesis. Advances in Neural Information Processing Systems, 34, 26609–26621.
  • Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems (pp. 2180–2188).
  • Cheng, Y., Wang, D., Zhou, P., & Zhang, T. (2020). A survey of model compression and acceleration for deep neural networks.
  • Choy, C. S., & Siu, W. (1998). A class of competitive learning models which avoids neuron underutilization problem. IEEE Transactions on Neural Networks, 9(6), 1258–1269. https://doi.org/10.1109/72.728374
  • Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. John Wiley and Sons, Inc.
  • da Cunha, A., Natale, E., & Viennot, L. (2022). Proving the strong lottery ticket hypothesis for convolutional neural networks. In International Conference on Learning Representations.
  • DeSieno, D. (1988). Adding a conscience to competitive learning. In IEEE International Conference on Neural Networks (Vol. 1, pp. 117–124).
  • Eckman, F. R., Bell, L., & Nelson, D. (1988). On the generalization of relative clause instruction in the acquisition of English as a second language. Applied Linguistics, 9(1), 1–20. https://doi.org/10.1093/applin/9.1.1
  • Erhan, D., Bengio, Y., Courville, A., & Vincent, P. (2009). Visualizing higher-layer features of a deep network. University of Montreal. 1341.
  • Fernández Rodríguez, J. D., Maza Quiroga, R. M., Palomo Ferrer, E. J., J. M. Ortiz-de-lazcano Lobato, & López-Rubio, E. (2022). A novel continual learning approach for competitive neural networks.
  • Frankle, J., & Carbin, M. (2018). The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635.
  • Frankle, J., Dziugaite, G. K., Roy, D., & Carbin, M. (2020). Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (pp. 3259–3269).
  • Frankle, J., Dziugaite, G. K., Roy, D. M., & Carbin, M. (2019). Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611.
  • Fritzke, B. (1993). Vector quantization with a growing and splitting elastic net. In Icann'93: International Conference on Artificial Neural Networks (pp. 580–585). Springer.
  • Fritzke, B. (1996). Automatic construction of radial basis function networks with the growing neural gas model and its relevance for fuzzy logic. In Applied computing: Proceedings of the Acm Symposium on Applied Computing (pp. 624–627). ACM.
  • Fukushima, K. (1975). Cognitron: A self-organizing multi-layered neural network. Biological Cybernetics, 20(3-4), 121–136. https://doi.org/10.1007/BF00342633
  • Grossberg, S. (1976). Adaptive pattern classification and universal recoding: I parallel development and coding of neural feature detectors. Biological Cybernetics, 23(3), 121–134. https://doi.org/10.1007/BF00344744
  • Gunasekar, S., Woodworth, B. E., Bhojanapalli, S., Neyshabur, B., & Srebro, N. (2017). Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 6152–6160.
  • Himberg, J. (2000). A SOM based cluster visualization and its application for false colouring. In Proceedings of the International Joint Conference on Neural Networks (pp. 69–74).
  • Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint, arXiv:1503.02531.
  • Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019). Adversarial examples are not bugs, they are features. arXiv preprint, arXiv:1905.02175.
  • Ishikawa, S. (2013). The ICNALE and sophisticated contrastive interlanguage analysis of Asian learners of English. Learner Corpus Studies in Asia and the World, 1(1), 91–118.
  • Izumi, S. (2003). Processing difficulty in comprehension and production of relative clauses by learners of English as a second language. Language Learning, 53(2), 285–323. https://doi.org/10.1111/1467-9922.00218
  • Kamimura, R. (2019). Neural self-compressor: Collective interpretation by compressing multi-layered neural networks into non-layered networks. Neurocomputing, 323(5), 12–36. https://doi.org/10.1016/j.neucom.2018.09.036.
  • Kohonen, T. (1990). The self-organizing maps. Proceedings of the IEEE, 78(9), 1464–1480. https://doi.org/10.1109/5.58325
  • Krotov, D., & Hopfield, J. J. (2019). Unsupervised learning by competing hidden units. Proceedings of the National Academy of Sciences, 116(16), 7723–7731. https://doi.org/10.1073/pnas.1820458116
  • Lagani, G., Falchi, F., Gennaro, C., & Amato, G. (2021). Training convolutional neural networks with competitive Hebbian learning approaches. In International Conference on Machine Learning, Optimization, and Data Science (pp. 25–40).
  • Lapuschkin, S., Binder, A., Montavon, G., Muller, K. R., & Samek, W. (2016). Analyzing classifiers: Fisher vectors and deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2912–2920).
  • Li, P., Tu, S., & Xu, L. (2022). Deep rival penalized competitive learning for low-resolution face recognition. Neural Networks.
  • Linsker, R. (1988). Self-organization in a perceptual network. Computer, 21(3), 105–117. https://doi.org/10.1109/2.36
  • Linsker, R. (1989). How to generate ordered maps by maximizing the mutual information between input and output signals. Neural Computation, 1(3), 402–411. https://doi.org/10.1162/neco.1989.1.3.402
  • Linsker, R. (1992). Local synaptic learning rules suffice to maximize mutual information in a linear network. Neural Computation, 4(5), 691–702. https://doi.org/10.1162/neco.1992.4.5.691
  • Lu, Y., Cheung, Y. M., & Tang, Y. Y. (2019). Self-adaptive multiprototype-based competitive learning approach: A k-means-type algorithm for imbalanced data clustering. IEEE Transactions on Cybernetics, 51(3), 1598–1612. https://doi.org/10.1109/TCYB.6221036
  • Luo, P., Zhu, Z., Liu, Z., Wang, X., & Tang, X. (2016). Face model compression by distilling knowledge from neurons. In Thirtieth aaai Conference on Artificial Intelligence.
  • Mahendran, A., & Vedaldi, A. (2015). Understanding deep image representations by inverting them. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5188–5196).
  • Makhzani, A., & Frey, B. (2014). A winner-take-all method for training sparse convolutional autoencoders. In Nips Deep Learning Workshop.
  • Malach, E., Yehudai, G., Shalev-Schwartz, S., & Shamir, O. (2020). Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning (pp. 6682–6691).
  • Mandt, S., Hoffman, M. D., & Blei, D. M. (2017). Stochastic gradient descent as approximate bayesian inference. arXiv preprint, arXiv:1704.04289.
  • McClelland, J. L, & Rumelhart, D. E. (1986). Parallel distributed processing. Vol. 1, MIT Press.
  • Montavon, G., Binder, A., Lapuschkin, S., Samek, W., & Müller, K. R. (2019). Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting, explaining and visualizing deep learning (pp. 193–209). Springer.
  • Nguyen, A., Yosinski, J., & Clune, J. (2019). Understanding neural networks via feature visualization: A survey. In Explainable AI: Interpreting, explaining and visualizing deep learning (pp. 55–76). Springer.
  • O'Grady, W., Lee, M., & Choo, M. (2003). A subject-object asymmetry in the acquisition of relative clauses in Korean as a second language. Studies in Second Language Acquisition, 25(3), 433–448. https://doi.org/10.1017/S0272263103000172
  • Papadopoulou, D., & Clahsen, H. (2003). Parsing strategies in L1 and L2 sentence processing: A study of relative clause attachment in Greek. Studies in Second Language Acquisition, 25(4), 501–528. https://doi.org/10.1017/S0272263103000214
  • Peng, B., Jin, L., & Shang, M. (2021). Multi-robot competitive tracking based on k-WTA neural network with one single neuron. Neurocomputing, 460(4), 1–8. https://doi.org/10.1016/j.neucom.2021.07.020.
  • Rumelhart, D. E., & McClelland, J. L. (1986). On learning the past tenses of English verbs. In D. E. Rumelhart, G. E. Hinton, & R. J. Williams, (Eds.), Parallel distributed processing (Vol. 2, pp. 216–271). MIT Press.
  • Rumelhart, D. E., & Zipser, D. (1985). Feature discovery by competitive learning. Cognitive Science, 9(1), 75–112. https://doi.org/10.1207/s15516709cog0901_5.
  • Shinozaki, T. (2017). Biologically inspired feedforward supervised learning for deep self-organizing map networks. arXiv preprint, arXiv:1710.09574.
  • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929–1958.
  • Srivastava, R. K., Masci, J., Kazerounian, S., Gomez, F., & Schmidhuber, J. (2013). Compete to compute. Advances in Neural Information Processing Systems, 26, 2312–2321.
  • Sturm, I., Lapuschkin, S., Samek, W., & Müller, K. R. (2016). Interpretable deep neural networks for single-trial EEG classification. Journal of Neuroscience Methods, 274(1), 141–145. https://doi.org/10.1016/j.jneumeth.2016.10.008.
  • Tian, Y., Jiang, T., Gong, Q., & Morcos, A. (2019). Luck matters: Understanding training dynamics of deep relu networks. arXiv preprint, arXiv:1905.13405.
  • Van Hulle, M. M. (1999). Faithful representations with topographic maps. Neural Networks, 12(6), 803–823. https://doi.org/10.1016/S0893-6080(99)00041-6
  • Van Hulle, M. M. (2004). Entropy-based kernel modeling for topographic map formation. IEEE Transactions on Neural Networks, 15(4), 850–858. https://doi.org/10.1109/TNN.2004.828763
  • Wakamatsu, H. (2018). Corpus analysis (in Japanese). In A. Hirai (Ed.), Data analysis for educational psycholinguistics. Tokyo Tosho.
  • Wang, L., Yan, Y., He, K., Wu, Y., & Xu, W. (2021). Dynamically disentangling social bias from task-oriented representations with adversarial attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3740–3750).
  • Wang, T., Zhao, J., Yatskar, M., Chang, K. W., & Ordonez, V. (2019). Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5310–5319).
  • Wu, L., Li, J., Wang, Y., Meng, Q., Qin, T., Chen, W., Zhang, M., & Liu, T. Y. (2021). R-drop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems, 34, 10890–10905.
  • Xia, Y., Zhou, J., Shi, Z., Lu, C., & Huang, H. (2020). Generative adversarial regularized mutual information policy gradient framework for automatic diagnosis. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, pp. 1062–1069).
  • Xiao, T., Li, H., Ouyang, W., & Wang, X. (2016). Learning deep feature representations with domain guided dropout for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1249–1258).
  • Xie, Q., Dai, Z., Du, Y., Hovy, E., & Neubig, G. (2017). Controllable invariance through adversarial feature learning. arXiv preprint, arXiv:1705.11122.
  • Xu, L., Xu, Y., & Chow, T. W. (2010). PolSOM: A new method for multidimensional data visualization. Pattern Recognition, 43(4), 1668–1675. https://doi.org/10.1016/j.patcog.2009.09.025
  • Yin, H. (2002). ViSOM-a novel method for multivariate data projection and structure visualization. Neural Networks, IEEE Transactions on, 13(1), 237–243. https://doi.org/10.1109/72.977314
  • Yu, Z., Guo, S., Deng, F., Yan, Q., Huang, K., Liu, J. K., & Chen, F. (2018). Emergent inference of hidden markov models in spiking neural networks through winner-take-all. IEEE Transactions on Cybernetics, 50(3), 1347–1354. https://doi.org/10.1109/TCYB.6221036
  • Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021). Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3), 107–115. https://doi.org/10.1145/3446776

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.