Introduction

An artificial neural network (ANN) is either a hardware implementation or a computer program which strives to simulate the information processing capabilities of its biological exemplar. ANNs are typically composed of a great number of interconnected artificial neurons. The artificial neurons are simplified models of their biological counterparts.

The typical characteristics of ANNs differ very much from what is normally expected of a computer. These new properties include adaptive learning, self-organization, error tolerance, real-time operation and parallel information processing.

Learning in the context of ANNs means, that the network can adopt different behavior on the basis of the data that is given to the network. Unlike telling the network how to react to each data vector separately, as would be the case in the conventional programming, the network itself is able to find properties from the presented data. The network learning can be continued as new data becomes available. Learning is thus adaptive.

As data is given to the ANN, it organizes its structure to reflect the properties of the given data. In most ANN models, the term self-organization refers to the determination of the connection strengths between neurons. The way the internal structure of an ANN is altered is determined by the used learning algorithm. Several distinct neural network models can be distinguished both from their internal architecture and from the learning algorithms that they use.

Error tolerance is an important aspect of an ANN. It refers to the network's ability to model the essential features of the given data. In other words, an ANN is capable of finding a generalization for the data. This powerful characteristic makes it possible to process new, imperfect and distorted data with neural networks.

Due to the parallel nature of the information processing in ANNs, real-time operation becomes possible. Basically, three entities characterize an ANN:

1. The network topology, or interconnection of neural 'units'
2. The characteristics of individual units or artificial neurons
3. The strategy for pattern learning or training

History of the Artificial Neural Networks

The history of the ANNs stems from the 1940s, the decade of the first electronic computer. However, the first significant step took place in 1957 when Rosenblatt introduced the first concrete neural model, the perceptron. Rosenblatt also took part in constructing the first successful neurocomputer, the Mark I Perceptron. After this initial impulse, the development of ANNs has proceeded as described in Figure 1.

Rosenblatt's original perceptron model contained only one layer. From this, a multi-layered model was derived in 1960. At first, the use of the multi-layer perceptron (MLP) was complicated by the lack of a suitable learning algorithm. In 1974, Werbos came to rescue by introducing a so-called backpropagation algorithm for the three-layered perceptron network. The application area of the MLP networks remained rather limited until the breakthrough in 1986 when a general backpropagation algorithm for a multi-layered perceptron was introduced by Rummelhart and Mclelland.

Hopfield brought out his idea of a neural network in 1982. Unlike the neurons in MLP, the Hopfield network consists of only one layer whose neurons are fully connected with each other. Since then, new versions of the Hopfield network have been developed. The Boltzmann machine has been influenced by both the Hopfield network and the MLP.

Adaptive Resonance Theory (ART) was first introduced by Carpenter and Grossberg in 1983. The development of ART has continued and resulted in the more advanced ART II and ART III network models.

Radial Basis Function (RBF) networks were first introduced by Broomhead & Lowe in 1988. Although the basic idea of RBF was developed 30 years ago under the name method of potential function, the work by Broomhead & Lowe opened a new frontier in the neural network community.

A totally unique kind of network model is the Self-Organizing Map (SOM) introduced by Kohonen in 1982. SOM is a certain kind of topological map which organizes itself based on the input patterns that it is trained with. The SOM originated from the LVQ (Learning Vector Quantization) network the underlying idea of which was also Kohonen's in 1972.

Figure 1: The evolution of the most popular artificial neural networks.

Structure of the ANN

The artificial neural networks can be classified according to the structure that they exhibit. Figure 2 represents four commonly used neural network structures.

Figure 2 a) represents the structure of a multi-layered feedforward network. The neurons in this ANN model are grouped in layers which are connected to the direction of the passing signal (from left to right in this case). There are no lateral connections within each layer and also no feedbackward connections within the network. The best-known ANN of this type is the perceptron network.

Figure 2 b) depicts a single-layered fully connected network model where each neuron is laterally connected to all neighbouring neurons in the layer. In this ANN model, all neurons are both input and output neurons. The best-known ANN of this type is the Hopfield network.

Figure 2 c) demonstrates the connections in a two-layered feedforward/feedbackward network. The layers in this ANN model are connected to both directions. As a pattern is presented to the network, it 'resonates' a certain number of times between the layers before a response is received from the output layer. The best-known ANN of this type is the Adaptive Resonance Theory (ART) network.

Figure 2 d) illustrates the idea of a topologically organized feature map. In this model, each neuron in the network contains a so-called feature vector. As a pattern from the training data is given to the network, the neuron whose feature vector is closest to the input vector is activated. The activated neuron is called the best matching unit (BMU) and it is updated to reflect input vector causing the activation. In the process of updating the BMU, the neighbouring neurons are updated towards the input vector or away from it (according to the learning algorithm in use). The network type exhibiting this kind of behaviour is the Self-Organizing Map of Kohonen.

Figure 2: Different ANN structures. a) Multi-layered feedforward network, b) single-layered fully connected network, c) two-layered feedforward/feedbackward network and d) topographically organized vector map.

Supervised vs. Unsupervised Learning

An important aspect of an ANN model is whether it needs guidance in learning or not. Based on the way they learn, all artificial neural networks can be divided into two learning categories - supervised and unsupervised.

In supervised learning, a desired output result for each input vector is required when the network is trained. An ANN of the supervised learning type, such as the multi-layer perceptron, uses the target result to guide the formation of the neural parameters. It is thus possible to make the neural network learn the behaviour of the process under study.

In unsupervised learning, the training of the network is entirely data-driven and no target results for the input data vectors are provided. An ANN of the unsupervised learning type, such as the self-organizing map, can be used for clustering the input data and find features inherent to the problem.

Application Areas of ANNs

Although a certain type of ANN has been engineered to address certain kinds of problems, there exist no definite rules as to what the exact application domains of ANNs are. The general application areas of ANNs are: robust pattern recognition, filtering, data segmentation, data compression, adaptive control, optimization, modelling complex functions and associative pattern recognition.

Figure 3 illustrates the use of well-known neural networks.
Figure 4 lists the application areas grouped according to the ANN structure.

Figure 3: Application areas of different neural networks.

Figure 4: Application areas of different ANNs grouped by network structure.