This question has often been debated since the beginnings of modern neuroscience but it is generally accepted that the average firing rate of neurons constitutes an important part of the neural code. In short, the classical view holds that mental entities are coded by cell assemblies (Hebb, 1949), which are spatial patterns of average activity.

Following Christoph von der Malsburg's "Correlation theory of brain function" (1981) and the work of my thesis advisor, Elie Bienenstock (see, e.g., von der Malsburg and Bienenstock, 1986), I defended another format of representation that involves higher-order moments or temporal correlations among neuronal activities. Here, mental representations are not exclusively based on individual mean activity rates <x_i>, which are events of order 1, but more generally on order-N events <x_i1 x_i2 ... x_iN> and, in particular, correlations between two neurons, <x_i x_j>.

Naturally, the traditional order-1 code stems from classical observations in the primary sensory areas (e.g., visual cortex), in which cells seem to possess selective response properties. From these experiments, it was inferred that one such neuron, or a small cluster of neurons, could individually and independently represent one specific type of stimulus (e.g., the orientation of an edge).

However, to obtain the global representation of an object these local features must eventually be linked and integrated. The problem is that this integration is unlikely to be carried out by highly specialized cells at the top of a hierarchical processing chain (the conjectural "grand-mother" cells that fire only when you see your grand-mother). Equally unlikely would be for the assembly of feature-coding activity rates to be maintained in a distributed state because of the impossiblity to overlap two such states without losing relational information (the so-called "binding problem"). If two cells coding for "red" and "circle" are active and two other cells coding for "green" and "triangle" also become active, then this global state of activation is undistinguishable from the alternative combination "red triangle" and "green circle" (von der Malsburg, 1987).

This is why we advocated the idea that feature integration requires higher-order codes to be able to represent relationships between elementary components that are initially uncorrelated (in the above example the spike trains of "red" and "circle" would be synchronous and out of phase with "green" and "triangle"). These correlation events bring to the representation format a structure that is fundamentally missing from the mere feature lists of Hebbian cell assemblies.

To use a chemical metaphor, we could say that feature lists are to molecular formulas (e.g., C₃H₈O) what correlations are to line-bond diagrams (e.g., 1-propanol vs. 2-propanol).

The first part does not offer a new method or algorithm but rather aims at bringing to light general problems and limitations encountered by statistical learning processes, especially of the generalist or "nonparametric" kind. The main goal of this study is to stress the crucial importance of identifying the right format of representation and giving it priority over other concerns about the "adaptability" or power of generalization of a learning system.

We put into practice this recommendation in the second part by designing a handwritten character classification method based on order-2 correlations. Images are represented by 2-D deformable lattices instead of unstructured lists of pixels, while the "distance" between two input images is defined as the cost-functional of a graph-matching process. The success rates achieved by this criteria are superior to feed-forward neural classifiers, implicitly based on Hamming or Euclidean metrics. This study illustrates both the importance of a problem-specific format of representation and the particular appropriateness of higher-order codes in this matter.

The third part is more speculative and attempts to answer Fodor and Pylyshyn's (1988) influential criticism about the lack of structured representations in neural networks. We show that the compositionality of language and cognition can actually arise from the simultaneous self-organization of connectivity and activity in an initially random cortical network. ← Less

Nonparametric estimation has been successful in numerous application domains and performs in some cases better than logical inference (expert systems, AI). However, it must also be asked whether this statistical approach is equally relevant to the cognitive and neurobiological domains. Does it provide an answer to the issue of neural representation? Can it solve complex cognitive problems such as invariant perception or language?

In short, when dealing with the nervous system and attempting to unravel the deep mechanisms of cognition, one may question whether the learning paradigm that drives the core of these methods is effectively as crucial as it is often claimed.

We show that the efficacy of any statistical estimator is bounded by universal limits. The mean quadratic error produced by an estimator (averaged over different learning data sets) is made of two terms, the bias and variance, which clearly point to the cause of these limitations. The bias term represents the average discrepancy between the system's predictions and the expected response, while the variance represents the fluctuations of these predictions with the various sample sets.

Our conclusion is that statistical estimation can yield good results in "simple" problems that do not require the system to extrapolate to unknown examples in a nontrivial way. Otherwise, if one expects a high power of generalization after the first interpolation phase, we claim that it becomes necessary to design into the system an adequate format of representation for the data and that the quality of this representation must have priority over tuning the learning parameters or selecting the examples. This is especially critical in "complex" problems of a cognitive nature.

This view is now widely accepted, yet it is still hoped in many cases that learning methods can actually discover by themselves the relevant data representation. In particular, these methods should be able to extract from the examples hidden component features that are characteristic of their class. This hope, again, has to be measured against the complexity of the problem. ← Less

Von der Malsburg's theory was prompted by the realization that the average rate-based conceptual format inevitably led to great difficulties or the impossibility to handle complex cognitive problems, including visual perception. In short, the classical Hebbian "cell assembly" format is lacking the structure necessary to code for relational information (see Introduction). Therefore, in this framework, the only way to represent relationships and avoid the "superposition catastrophe" (see the "red circle/green triangle" example of the introduction) is to dedicate new cells for each new combination of features.

This solution is obviously not realistic and it is much more natural to code relations with temporal correlations among cellular activities rather than with new cells. Moreover, through a Hebbian-like positive feedback loop on the millisecond timescale, these correlations reinforce the synaptic connections that support them. Therefore, instead of correlations, a relational structure can be equivalently represented by dynamical links.

The cost-functional contains two constraint terms, one penalizing the elastic deformation of edges between neighboring nodes, the other penalizing mismatches between the labels carried by pairs of mapped nodes. The best mapping is thus a trade-off between structural integrity and feature integrity.

We carried out numerical experiments on a public database of 1200 16x16-pixel handwritten digits and compared the results with traditional feed-forward layered network methods. Defining our own graph-matching cost-functional as a "pseudo-metric" in image space, we applied a simple nearest-neighbor decision criterion and found significantly lower error rates. This proved that designing an appropriate representation format (here, 2-D relational) is often more important than fine-tuning the parameters of a learning process.

I later created a new version of 2-D graph matching based on temporal correlations and phase-locking in spiking neural networks (Doursat et al. 1995). ← Less

In this context, the issue of an appropriate format of representation of mental entities is of particular importance and our proposal is that the nervous system uses a higher-order code to represent linguistic entities.

The goal of the present neural model was to show that compositionality can arise from the gradual ontogenetic development of the nervous system during the early stages of synaptogenesis. By this, we adhered to Chomsky's conception that language actually "grows" and matures in children's brain like a limb or an organ. This claim might sound suprising at first but is in accordance with well-known observations and general principles of neural development.

The visual system and many other cortical areas display striking regularities in their connectivity, which self-organize during fetal and postnatal development (with or without input from external stimuli) and account for their functional specialization. Similarly, it is postulated here that the faculty of language (as opposed to any specific language) is supported by specialized neural pathways that develop through a feedback interaction between neuronal activities and synaptic efficacies.

Starting from an initially disordered network with low random activity, certain synaptic connections are gradually selected and strengthened to the detriment of others. This focusing of the connectivity is also accompanied by a gradual increase and durability of correlated firing. Connections and correlations reinforce each other through heterosynaptic cooperation, while the global stability of the network is maintained through a constraint of competition.

On the whole, complex spatiotemporal patterns of spiking activity spontaneously emerge within the network. These patterns have been experimentally detected in mammalian species and termed synfire chains (Abeles 1982). In our claim, they constitute the elementary components or building blocks of compositionality. Such patterns have the required properties discussed above, i.e., an internal combinatorial structure that allows them to assemble in multiple ways, thereby opening the way to a virtually infinite hierarchy of combinations.

For a more recent version of this study, see the SYNDEVO project. ← Less