Flexible feature discovery and structural information control

In this paper, we propose a new information theoretic method called structural information control for flexible feature discovery. The new method has three distinctive characteristics, which traditional competitive learning fails to offer. First, the new method can directly control competitive unit activation patterns, whereas traditional competitive learning does not have any means to control them. Thus, with the new method, it is possible to extract salient features not discovered by traditional methods. Second, competitive units compete witheach other by maximizing their information content about input patterns. Consequently, this information maximization makes it possible to control flexibly competition processes. Third, in structural information control, it is possible to define many different kinds of information content, and we can choose a specific type of information according to a given objective. When applied to competitive learning, structural information can be used to control the number of dead or spare units, and to extract macro as well as micro features of input patterns in explicit ways. We first applied this method to simple pattern classification to demonstrate that information can be controlled and that different neuron firing patterns can be generated. Second, a dipole problem was used to show that structural information could provide representations similar to those by the conventional competitive learning methods. Finally, we applied the method to a language acquisition problem in which networks must flexibly discover some linguistic rules by changing structural information. Especially, we attempted to examine the effect of the information parameter to control the number of dead neurons, and thus to examine how macro and micro features in input patterns can explicitly be discovered by structural information.