Artificial Neural Network for Biomedical Purpose

ARTIFICIAL NEURAL NETWORKS METHODOLOGICAL ADVANCES AND BIOMEDICAL APPLICATIONS
Edited by Kenji Suzuki

Artificial Neural Networks - Methodological Advances and Biomedical Applications Edited by Kenji Suzuki Published by InTech Janeza Trdine 9, 51000 Rijeka, Croatia Copyright © 2011 InTech All chapters are Open Access articles distributed under the Creative Commons Non Commercial Share Alike Attribution 3.0 license, which permits to copy, distribute, transmit, and adapt the work in any medium, so long as the original work is properly cited. After this work has been published by InTech, authors have the right to republish it, in whole or part, in any publication of which they are the author, and to make other personal use of the work. Any republication, referencing or personal use of the work must explicitly identify the original source. Statements and opinions expressed in the chapters are these of the individual contributors and not necessarily those of the editors or publisher. No responsibility is accepted for the accuracy of information contained in the published articles. The publisher assumes no responsibility for any damage or injury to persons or property arising out of the use of any materials, instructions, methods or ideas contained in the book. Publishing Process Manager Ivana Lorkovic Technical Editor Teodora Smiljanic Cover Designer Martina Sirotic Image Copyright Bruce Rolff, 2010. Used under license from Shutterstock.com First published March, 2011 Printed in India A free online edition of this book is available at www.intechopen.com Additional hard copies can be obtained from orders@intechweb.org

Artificial Neural Networks - Methodological Advances and Biomedical Applications Edited by Kenji Suzuki p. cm. ISBN 978-953-307-243-2

free online editions of InTech Books and Journals can be found at www.intechopen.com

Contents
Preface Part 1 Chapter 1 IX 1

Fundamentals

Introduction to the Artificial Neural Networks 3 Andrej Krenker, Janez Bešter and Andrej Kos Review of Input Variable Selection Methods for Artificial Neural Networks 19 Robert May, Graeme Dandy and Holger Maier Artificial Neural Networks and Efficient Optimization Techniques for Applications in Engineering 45 Rossana M. S. Cruz, Helton M. Peixoto and Rafael M. Magalhães Advanced Architectures for Biomedical Applications 69 Pixel-Based Artificial Neural Networks in Computer-Aided Diagnosis 71 Kenji Suzuki Applied Artificial Neural Networks: from Associative Memories to Biomedical Applications 93 Mahmood Amiri and Katayoun Derakhshandeh Medical Image Segmentation Using Artificial Neural Networks 121 Mostafa Jabarouti Moghaddam and Hamid Soltanian-Zadeh Artificial Neural Networks and Predictive Medicine: a Revolutionary Paradigm Shift 139 Enzo Grossi Reputation-Based Neural Network Combinations Mohammad Nikjoo, Azadeh Kushki, Joon Lee, Catriona Steele and Tom Chau 151

Chapter 2

Chapter 3

Part 2 Chapter 4

Chapter 5

Chapter 6

Chapter 7

Chapter 8

VI

Contents

Part 3 Chapter 9

Biological Applications

171

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research 173 Norio Kobayashi and Tetsuro Toyoda Artificial Neural Networks Technology to Model and Predict Plant Biology Process 197 Pedro P. Gallego, Jorge Gago and Mariana Landín The Usefulness of Artificial Neural Networks in Predicting the Outcome of Hematopoietic Stem Cell Transplantation 217 Giovanni Caocci, Roberto Baccoli and Giorgio La Nasa Artificial Neural Networks and Retinal Ganglion Cell Responses 233 María P. Bonomini, José M. Ferrández and Eduardo Fernández Medical Applications 251

Chapter 10

Chapter 11

Chapter 12

Part 4 Chapter 13

Diagnosing Skin Diseases Using an Artificial Neural Network Bakpo, F. S. and Kabari, L. G

253

Chapter 14

Artificial Neural Networks Used to Study the Evolution of the Multiple Sclerosis Tabares Ospina and Hector Anibal

271

Chapter 15

Estimation the Depth of Anesthesia by the Use of Artificial Neural Network 283 Hossein Rabbani, Alireza Mehri Dehnavi and Mehrab Ghanatbari Artificial Neural Networks (ANN) Applied for Gait Classification and Physiotherapy Monitoring in Post Stroke Patients 303 Katarzyna Kaczmarczyk, Andrzej Wit, Maciej Krawczyk and Jacek Zaborski Clinical and Other Applications 329

Chapter 16

Part 5 Chapter 17

Forcasting the Clinical Outcome: Artificial Neural Networks or Multivariate Statistical Models? 331 Ahmed Akl and Mohamed A Ghoneim Telecare Adoption Model Based on Artificial Neural Networks 343 Jui-Chen Huang

Chapter 18

Contents

VII

Chapter 19

Effectiveness of Artificial Neural Networks in Forecasting Failure Risk for Pre-Medical Students 355 Jawaher K. Alenezi, Mohammed M. Awny and Maged M. M. Fahmy

Preface
Articial neural networks may probably be the single most successful technology in the last two decades which has been widely used in a large variety of applications in various areas. An articial neural network, often just called a neural network, is a mathematical (or computational) model that is inspired by the structure and function of biological neural networks in the brain. An articial neural network consists of a number of articial neurons (i.e., nonlinear processing units) which are connected each other via synaptic weights (or simply just weights). An articial neural network can “learn” a task by adjusting weights. There are supervised and unsupervised models. A supervised model requires a “teacher” or desired (ideal) output to learn a task. An unsupervised model does not require a “teacher,” but it leans a task based on a cost function associated with the task. An articial neural network is a powerful, versatile tool. Articial neural networks have been successfully used in various applications such as biological, medical, industrial, control engendering, software engineering, environmental, economical, and social applications. The high versatility of articial neural networks comes from its high capability and learning function. It has been theoretically proved that an articial neural network can approximate any continuous mapping by arbitrary precision. Desired continuous mapping or a desired task is acquired in an articial neural network by learning. The purpose of this book series is to provide recent advances of articial neural network applications in a wide range of areas. The series consists of two volumes: the rst volume contains methodological advances and biomedical applications of articial neural networks; the second volume contains articial neural network applications in industrial and control engineering. This rst volume begins with a section of fundamentals of articial neural networks which covers an introduction, design, and optimization of articial neural networks. The fundamental concept, principles, and theory in the section help understand and use an articial neural network in a specic application properly as well as e ectively. A section of advanced architectures for biomedical applications follows. Researchers have developed advanced architectures for articial neural networks specically for biomedical applications. Such advanced architectures o er improved performance and desirable properties. Sections continue with biological applications such as gene, plant biology, and stem cell, medical applications such as skin diseases, sclerosis, anesthesia, and physiotherapy, and clinical and other applications such as clinical outcome, telecare, and pre-med student failure prediction.

X

Preface

Thus, this book will be a fundamental source of recent advances and applications of articial neural networks in biomedical areas. The target audience of this book includes professors, college students, and graduate students in engineering and medical schools, engineers in biomedical companies, researchers in biomedical and health sciences, medical doctors such as radiologists, cardiologists, pathologists, and surgeons, healthcare professionals such as radiology technologists and medical physicists. I hope this book will be a useful source for readers and inspire them.

Kenji Suzuki, Ph.D. University of Chicago Chicago, Illinois, USA

Part 1
Fundamentals

1
Introduction to the Artificial Neural Networks
Andrej Krenker1, Janez Bešter2 and Andrej Kos2
2Faculty 1Consalta d.o.o. of Electrical Engineering, University of Ljubljana Slovenia

1. Introduction
An Artificial Neural Network (ANN) is a mathematical model that tries to simulate the structure and functionalities of biological neural networks. Basic building block of every artificial neural network is artificial neuron, that is, a simple mathematical model (function). Such a model has three simple sets of rules: multiplication, summation and activation. At the entrance of artificial neuron the inputs are weighted what means that every input value is multiplied with individual weight. In the middle section of artificial neuron is sum function that sums all weighted inputs and bias. At the exit of artificial neuron the sum of previously weighted inputs and bias is passing trough activation function that is also called transfer function (Fig. 1.).

Fig. 1. Working principle of an artificial neuron.

4

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Although the working principles and simple set of rules of artificial neuron looks like nothing special the full potential and calculation power of these models come to life when we start to interconnect them into artificial neural networks (Fig. 2.). These artificial neural networks use simple fact that complexity can grown out of merely few basic and simple rules.

Fig. 2. Example of simple artificial neural network. In order to fully harvest the benefits of mathematical complexity that can be achieved through interconnection of individual artificial neurons and not just making system complex and unmanageable we usually do not interconnect these artificial neurons randomly. In the past, researchers have come up with several “standardised” topographies of artificial neural networks. These predefined topographies can help us with easier, faster and more efficient problem solving. Different types of artificial neural network topographies are suited for solving different types of problems. After determining the type of given problem we need to decide for topology of artificial neural network we are going to use and then fine-tune it. We need to fine-tune the topology itself and its parameters. Fine tuned topology of artificial neural network does not mean that we can start using our artificial neural network, it is only a precondition. Before we can use our artificial neural network we need to teach it solving the type of given problem. Just as biological neural networks can learn their behaviour/responses on the basis of inputs that they get from their environment the artificial neural networks can do the same. There are three major learning paradigms: supervised learning, unsupervised learning and reinforcement learning. We choose learning paradigm similar as we chose artificial neuron network topography - based on the problem we are trying to solve. Although learning paradigms are different in their principles they all have one thing in common; on the basis of “learning data” and “learning rules” (chosen cost function) artificial neural network is trying to achieve proper output response in accordance to input signals. After choosing topology of an artificial neural network, fine-tuning of the topology and when artificial neural network has learn a proper behaviour we can start using it for solving given problem. Artificial neural networks have been in use for some time now and we can find them working in areas such as process control, chemistry, gaming, radar systems, automotive industry, space industry, astronomy, genetics, banking, fraud detection, etc. and solving of problems like function approximation, regression analysis, time series prediction, classification, pattern recognition, decision making, data processing, filtering, clustering, etc., naming a few.

Introduction to the Artificial Neural Networks

5

As topic of artificial neural networks is complex and this chapter is only informative nature we encourage novice reader to find detail information on artificial neural networks in (Gurney, 1997; Kröse & Smagt 1996; Paveši , 2000; Rojas 1996).

2. Artificial neuron
Artificial neuron is a basic building block of every artificial neural network. Its design and functionalities are derived from observation of a biological neuron that is basic building block of biological neural networks (systems) which includes the brain, spinal cord and peripheral ganglia. Similarities in design and functionalities can be seen in Fig.3. where the left side of a figure represents a biological neuron with its soma, dendrites and axon and where the right side of a figure represents an artificial neuron with its inputs, weights, transfer function, bias and outputs.

Fig. 3. Biological and artificial neuron design. In case of biological neuron information comes into the neuron via dendrite, soma processes the information and passes it on via axon. In case of artificial neuron the information comes into the body of an artificial neuron via inputs that are weighted (each input can be individually multiplied with a weight). The body of an artificial neuron then sums the weighted inputs, bias and “processes” the sum with a transfer function. At the end an artificial neuron passes the processed information via output(s). Benefit of artificial neuron model simplicity can be seen in its mathematical description below: (1) Where: is input value in discrete time where goes from to , is weight value in discrete time where goes from to , is bias, is a transfer function, is output value in discrete time . As seen from a model of an artificial neuron and its equation (1) the major unknown variable of our model is its transfer function. Transfer function defines the properties of artificial neuron and can be any mathematical function. We choose it on the basis of problem that artificial neuron (artificial neural network) needs to solve and in most cases we choose it from the following set of functions: Step function, Linear function and Non-linear (Sigmoid) function.

6

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Step function is binary function that has only two possible output values (e.g. zero and one). That means if input value meets specific threshold the output value results in one value and if specific threshold is not meet that results in different output value. Situation can be described with equation (2). (2) When this type of transfer function is used in artificial neuron we call this artificial neuron perceptron. Perceptron is used for solving classification problems and as such it can be most commonly found in the last layer of artificial neural networks. In case of linear transfer function artificial neuron is doing simple linear transformation over the sum of weighted inputs and bias. Such an artificial neuron is in contrast to perceptron most commonly used in the input layer of artificial neural networks. When we use non-linear function the sigmoid function is the most commonly used. Sigmoid function has easily calculated derivate, which can be important when calculating weight updates in the artificial neural network.

3. Artificial Neural Networks
When combining two or more artificial neurons we are getting an artificial neural network. If single artificial neuron has almost no usefulness in solving real-life problems the artificial neural networks have it. In fact artificial neural networks are capable of solving complex real-life problems by processing information in their basic building blocks (artificial neurons) in a non-linear, distributed, parallel and local way. The way that individual artificial neurons are interconnected is called topology, architecture or graph of an artificial neural network. The fact that interconnection can be done in numerous ways results in numerous possible topologies that are divided into two basic classes. Fig. 4. shows these two topologies; the left side of the figure represent simple feedforward topology (acyclic graph) where information flows from inputs to outputs in only one direction and the right side of the figure represent simple recurrent topology (semicyclic graph) where some of the information flows not only in one direction from input to output but also in opposite direction. While observing Fig. 4. we need to mention that for easier handling and mathematical describing of an artificial neural network we group individual neurons in layers. On Fig. 4. we can see input, hidden and output layer.

Fig. 4. Feed-forward (FNN) and recurrent (RNN) topology of an artificial neural network.

Introduction to the Artificial Neural Networks

7

When we choose and build topology of our artificial neural network we only finished half of the task before we can use this artificial neural network for solving given problem. Just as biological neural networks need to learn their proper responses to the given inputs from the environment the artificial neural networks need to do the same. So the next step is to learn proper response of an artificial neural network and this can be achieved through learning (supervised, un-supervised or reinforcement learning). No matter which method we use, the task of learning is to set the values of weight and biases on basis of learning data to minimize the chosen cost function. 3.1 Feed-forward Artificial Neural Networks Artificial neural network with feed-forward topology is called Feed-Forward artificial neural network and as such has only one condition: information must flow from input to output in only one direction with no back-loops. There are no limitations on number of layers, type of transfer function used in individual artificial neuron or number of connections between individual artificial neurons. The simplest feed-forward artificial neural network is a single perceptron that is only capable of learning linear separable problems. Simple multi-layer feed-forward artificial neural network for purpose of analytical description (sets of equations (3), (4) and (5)) is shown on Fig. 5. (3)

(4)

(5)

Fig. 5. Feed-forward artificial neural network.

8

Artificial Neural Networks - Methodological Advances and Biomedical Applications

As seen on Fig. 5 and corresponding analytical description with sets of equations (3), (4) and (5) the simple feed-forward artificial neural network can led to relatively long mathematical descriptions where artificial neural networks’ parameters optimization problem solving by hand is impractical. Although analytical description can be used on any complex artificial neural network in practise we use computers and specialised software that can help us build, mathematically describe and optimise any type of artificial neural network. 3.2 Recurrent Artificial Neural Networks Artificial neural network with the recurrent topology is called Recurrent artificial neural network. It is similar to feed-forward neural network with no limitations regarding backloops. In these cases information is no longer transmitted only in one direction but it is also transmitted backwards. This creates an internal state of the network which allows it to exhibit dynamic temporal behaviour. Recurrent artificial neural networks can use their internal memory to process any sequence of inputs. Fig. 6. shows small Fully Recurrent artificial neural network and complexity of its artificial neuron interconnections. The most basic topology of recurrent artificial neural network is fully recurrent artificial network where every basic building block (artificial neuron) is directly connected to every other basic building block in all direction. Other recurrent artificial neural networks such as Hopfield, Elman, Jordan, bi-directional and other networks are just special cases of recurrent artificial neural networks.

Fig. 6. Fully recurrent artificial neural network. 3.3 Hopfield Artificial Neural Network A Hopfield artificial neural network is a type of recurrent artificial neural network that is used to store one or more stable target vectors. These stable vectors can be viewed as memories that the network recalls when provided with similar vectors that act as a cue to the network memory. These binary units only take two different values for their states that are determined by whether or not the units' input exceeds their threshold. Binary units can take either values of 1 or -1, or values of 1 or 0. Consequently there are two possible definitions for binary unit activation (equation (6) and (7)):

Introduction to the Artificial Neural Networks

9 (6)

(7) Where: is the strength of the connection weight from unit j to unit i, is the state of unit j, is the threshold of unit i. While talking about connections we need to mention that there are typical two restrictions: no unit has a connection with itself ( ) and that connections are symmetric . The requirement that weights must be symmetric is typically used, as it will guarantee that the energy function decreases monotonically while following the activation rules. If nonsymmetric weights are used the network may exhibit some periodic or chaotic behaviour. Training a Hopfield artificial neural network (Fig. 7.) involves lowering the energy of states that the artificial neural network should remember.

Fig. 7. Simple “one neuron” Hopfield artificial neural network. 3.4 Elman and Jordan Artificial Neural Networks Elman network also referred as Simple Recurrent Network is special case of recurrent artificial neural networks. It differs from conventional two-layer networks in that the first layer has a recurrent connection. It is a simple three-layer artificial neural network that has back-loop from hidden layer to input layer trough so called context unit (Fig. 8.). This type of artificial neural network has memory that allowing it to both detect and generate time-varying patterns. The Elman artificial neural network has typically sigmoid artificial neurons in its hidden layer, and linear artificial neurons in its output layer. This combination of artificial neurons transfer functions can approximate any function with arbitrary accuracy if only there is enough artificial neurons in hidden layer. Being able to store information Elman artificial neural network is capable of generating temporal patterns as well as spatial patterns and

10

Artificial Neural Networks - Methodological Advances and Biomedical Applications

responding on them. Jordan network (Fig. 9.) is similar to Elman network. The only difference is that context units are fed from the output layer instead of the hidden layer.

Fig. 8. Elman artificial neural network.

Fig. 9. Jordan artificial neural network. 3.5 Long Short Term Memory Long Short Term Memory is one of the recurrent artificial neural networks topologies. In contrast with basic recurrent artificial neural networks it can learn from its experience to process, classify and predict time series with very long time lags of unknown size between important events. This makes Long Short Term Memory to outperform other recurrent artificial neural networks, Hidden Markov Models and other sequence learning methods. Long Short Term Memory artificial neural network is build from Long Short Term Memory blocks that are capable of remembering value for any length of time. This is achieved with gates that determine when the input is significant enough remembering it, when continue to remembering or forgetting it, and when to output the value. Architecture of Long Short Term Memory block is shown in Fig. 10 where input layer consists of sigmoid units. Top neuron in the input layer process input value that might be

Introduction to the Artificial Neural Networks

11

sent to a memory unit depends on computed value of second neuron from the top in the input layer. The third neuron from the top in the input layer decide how long will memory unit hold (remember) its value and the bottom most neuron determines when value from memory should be released to the output. Neurons in first hidden layer and in output layer are doing simple multiplication of their inputs and a neuron in the second hidden layer computes simple linear function of its inputs. Output of the second hidden layer is fed back into input and first hidden layer in order to help making decisions.

Fig. 10. Simple Long Short Term Memory artificial neural network (block). 3.6 Bi-directional Artificial Neural Networks (Bi-ANN) Bi-directional artificial neural networks (Fig. 11.) are designed to predict complex time series. They consist of two individual interconnected artificial neural (sub) networks that performs direct and inverse (bidirectional) transformation. Interconnection of artificial neural sub networks is done through two dynamic artificial neurons that are capable of remembering their internal states. This type of interconnection between future and past values of the processed signals increase time series prediction capabilities. As such these artificial neural networks not only predict future values of input data but also past values. That brings need for two phase learning; in first phase we teach one artificial neural sub network for predicting future and in the second phase we teach a second artificial neural sub network for predicting past. 3.7 Self-Organizing Map (SOM) Self-organizing map is an artificial neural network that is related to feed-forward networks but it needs to be told that this type of architecture is fundamentally different in arrangement of neurons and motivation. Common arrangement of neurons is in a hexagonal or rectangular grid (Fig. 12.). Self-organizing map is different in comparison to other artificial neural networks in the sense that they use a neighbourhood function to preserve the topological properties of the input space. They uses unsupervised learning paradigm to

12

Artificial Neural Networks - Methodological Advances and Biomedical Applications

produce a low-dimensional, discrete representation of the input space of the training samples, called a map what makes them especially useful for visualizing low-dimensional views of high-dimensional data. Such networks can learn to detect regularities and correlations in their input and adapt their future responses to that input accordingly.

Fig. 11. Bi-directional artificial neural network.

Fig. 12. Self-organizing Map in rectangular (left) and hexagonal (right) grid.

Introduction to the Artificial Neural Networks

13

Just as others artificial neural networks need learning before they can be used the same goes for self-organizing map; where the goal of learning is to cause different parts of the artificial neural network to respond similarly to certain input patterns. While adjusting the weights of the neurons in the process of learning they are initialized either to small random values or sampled evenly from the subspace spanned by the two largest principal component eigenvectors. After initialization artificial neural network needs to be fed with large number of example vectors. At that time Euclidean distance to all weight vectors is computed and the neuron with weight vector most similar to the input is called the best matching unit. The weights of the best matching unit and neurons close to it are adjusted towards the input vector. This process is repeated for each input vector for a number of cycles. After learning phase we do so-called mapping (usage of artificial neural network) and during this phase the only one neuron whose weight vector lies closest to the input vector will be winning neuron. Distance between input and weight vector is again determined by calculating the Euclidean distance between them. 3.8 Stochastic Artificial Neural Network Stochastic artificial neural networks are a type of an artificial intelligence tool. They are built by introducing random variations into the network, either by giving the network's neurons stochastic transfer functions, or by giving them stochastic weights. This makes them useful tools for optimization problems, since the random fluctuations help it escape from local minima. Stochastic neural networks that are built by using stochastic transfer functions are often called Boltzmann machine. 3.9 Physical Artificial Neural Network Most of the artificial neural networks today are software-based but that does not exclude the possibility to create them with physical elements which base on adjustable electrical current resistance materials. History of physical artificial neural networks goes back in 1960’s when first physical artificial neural networks were created with memory transistors called memistors. Memistors emulate synapses of artificial neurons. Although these artificial neural networks were commercialized they did not last for long due to their incapability for scalability. After this attempt several others followed such as attempt to create physical artificial neural network based on nanotechnology or phase change material.

4. Learning
There are three major learning paradigms; supervised learning, unsupervised learning and reinforcement learning. Usually they can be employed by any given type of artificial neural network architecture. Each learning paradigm has many training algorithms. 4.1 Supervised learning Supervised learning is a machine learning technique that sets parameters of an artificial neural network from training data. The task of the learning artificial neural network is to set the value of its parameters for any valid input value after having seen output value. The training data consist of pairs of input and desired output values that are traditionally represented in data vectors. Supervised learning can also be referred as classification, where we have a wide range of classifiers, each with its strengths and weaknesses. Choosing a

14

Artificial Neural Networks - Methodological Advances and Biomedical Applications

suitable classifier (Multilayer perceptron, Support Vector Machines, k-nearest neighbour algorithm, Gaussian mixture model, Gaussian, naive Bayes, decision tree, radial basis function classifiers,…) for a given problem is however still more an art than a science. In order to solve a given problem of supervised learning various steps has to be considered. In the first step we have to determine the type of training examples. In the second step we need to gather a training data set that satisfactory describe a given problem. In the third step we need to describe gathered training data set in form understandable to a chosen artificial neural network. In the fourth step we do the learning and after the learning we can test the performance of learned artificial neural network with the test (validation) data set. Test data set consist of data that has not been introduced to artificial neural network while learning. 4.2 Unsupervised learning Unsupervised learning is a machine learning technique that sets parameters of an artificial neural network based on given data and a cost function which is to be minimized. Cost function can be any function and it is determined by the task formulation. Unsupervised learning is mostly used in applications that fall within the domain of estimation problems such as statistical modelling, compression, filtering, blind source separation and clustering. In unsupervised learning we seek to determine how the data is organized. It differs from supervised learning and reinforcement learning in that the artificial neural network is given only unlabeled examples. One common form of unsupervised learning is clustering where we try to categorize data in different clusters by their similarity. Among above described artificial neural network models, the Self-organizing maps are the ones that the most commonly use unsupervised learning algorithms. 4.3 Reinforcement learning Reinforcement learning is a machine learning technique that sets parameters of an artificial neural network, where data is usually not given, but generated by interactions with the environment. Reinforcement learning is concerned with how an artificial neural network ought to take actions in an environment so as to maximize some notion of long-term reward. Reinforcement learning is frequently used as a part of artificial neural network’s overall learning algorithm. After return function that needs to be maximized is defined, reinforcement learning uses several algorithms to find the policy which produces the maximum return. Naive brute force algorithm in first step calculates return function for each possible policy and chooses the policy with the largest return. Obvious weakness of this algorithm is in case of extremely large or even infinite number of possible policies. This weakness can be overcome by value function approaches or direct policy estimation. Value function approaches attempt to find a policy that maximizes the return by maintaining a set of estimates of expected returns for one policy; usually either the current or the optimal estimates. These methods converge to the correct estimates for a fixed policy and can also be used to find the optimal policy. Similar as value function approaches the direct policy estimation can also find the optimal policy. It can find it by searching it directly in policy space what greatly increases the computational cost. Reinforcement learning is particularly suited to problems which include a long-term versus short-term reward trade-off. It has been applied successfully to various problems, including

Introduction to the Artificial Neural Networks

15

robot control, telecommunications, and games such as chess and other sequential decision making tasks.

5. Usage of Artificial Neural Networks
One of the greatest advantages of artificial neural networks is their capability to learn from their environment. Learning from the environment comes useful in applications where complexity of the environment (data or task) make implementations of other type of solutions impractical. As such artificial neural networks can be used for variety of tasks like classification, function approximation, data processing, filtering, clustering, compression, robotics, regulations, decision making, etc. Choosing the right artificial neural network topology depends on the type of the application and data representation of a given problem. When choosing and using artificial neural networks we need to be familiar with theory of artificial neural network models and learning algorithms. Complexity of the chosen model is crucial; using to simple model for specific task usually results in poor or wrong results and over complex model for a specific task can lead to problems in the process of learning. Complex model and simple task results in memorizing and not learning. There are many learning algorithms with numerous tradeoffs between them and almost all are suitable for any type of artificial neural network model. Choosing the right learning algorithm for a given task takes a lot of experiences and experimentation on given problem and data set. When artificial neural network model and learning algorithm is properly selected we get robust tool for solving given problem. 5.1 Example: Using bi-directional artificial neural network for ICT fraud detection Spread of Information and Communication Technologies results in not only benefits for individuals and society but also in threats and increase of Information and Communication Technology frauds. One of the main tasks for Information and Communication Technology developers is to prevent potential fraudulent misuse of new products and services. If protection against fraud fails there is a vital need to detect frauds as soon as possible. Information and Communication Technology frauds detection is based on numerous principles. One of such principle is use of artificial neural networks in the detection algorithms. Below is an example of how to use bi-directional artificial neural network for detecting mobile-phone fraud. First task is to represent problem of detecting our fraud in the way that can be easily understand by humans and machines (computers). Each individual user or group of users behave in specific way while using mobile phone. By learning their behaviour we can teach our system to recognize and predict users’ future behaviour to a certain degree of accuracy. Later comparison between predicted and real-life behaviour and potential discrepancy between them can indicate a potential fraudulent behaviour. It was shown that mobilephone usage behaviour can be represented in the form of time series suitable for further analysis with artificial neural networks (Krenker et al., 2009). With this representation we transform the behaviour prediction task in time series prediction task. Time series prediction task can be realized with several different types of artificial neural networks but as mentioned in earlier chapters some are more suitable then others. Because we expect long and short time periods between important events in our data representation of users’ behaviour the most obvious artificial neural networks to use are Long Short Term Memory and bi-directional

16

Artificial Neural Networks - Methodological Advances and Biomedical Applications

artificial neural networks. On the basis of others researchers’ favourable results in time series prediction with bi-directional artificial neural network (Wakuya & Shida, 2001) we decided to use this artificial neural network topology for predicting our time series. After we choose artificial neural network architecture we choose the type of learning paradigm; we choose supervised learning where we gather real life data form telecommunication system. Gathered data was divided into two sub-sets; training sub-set and validation subset. With training data sub-set artificial neural network learn to predict future and past time series and with validation data sub-set we simulate and validate the prediction capabilities of designed and fine-tuned bi-directional artificial neural networks. Validation was done with calculation of the Average Relative Variance that represents a measure of similarity between predicted and expected time series. Only after we gathered information about mobile-phone fraud and after choosing representation of our problem and basic approaches for solving it we could start building the overall model for detecting mobile-phone fraud (Fig. 13.). On Fig. 13. we can see that mobile-phone fraud detection model is build out of three modules; input module, artificial neural network module and comparison module. Input Module gathers users’ information about usage of mobile-phone from telecommunication system in three parts. In first part it is used for gathering learning data from which Artificial Neural Network Module learn it-self. In second part Input Module gathers users’ data for purpose of validating the Artificial Neural Network Module and in the third part it collects users’ data in real time for purpose of using deployed mobile-phone fraud system. Artificial Neural Network Module is bidirectional artificial neural network that is learning from gathered data and later when the mobile-phone fraud detection system is deployed continuously predicts time series that represents users’ behaviour. Comparison module is used for validation of Artificial Neural Network Module in the process of learning and later when the mobile-phone fraud detection system is deployed it is used for triggering alarms in case of discrepancies between predicted and real-life gathered information about users’ behaviour.

Fig. 13. Mobile-phone fraud detection model.

Introduction to the Artificial Neural Networks

17

Although mobile-phone fraud detection system described above is simple and straight forward reader needs to realize that majority of work is not in creating and later implementing desired systems but in fine-tuning of data representation and artificial neural network architecture and its parameters that is strongly dependant on type of input data.

6. Conclusions
Artificial neural networks are widely spread and used in everyday services, products and applications. Although modern software products enable relatively easy handling with artificial neural networks, their creation, optimisation and usage in real-life situations it is necessary to understand theory that stands behind them. This chapter of the book introduces artificial neural networks to novice reader and serves as a stepping stone for all of those who would like to get more involved in the area of artificial neural networks. In the Introduction in order to lighten the area of artificial neural networks we briefly described basic building blocks (artificial neuron) of artificial neural networks and their “transformation” from single artificial neuron to complete artificial neural network. In the chapter Artificial Neuron we present basic and important information about artificial neuron and where researchers borrowed the idea to create one. We show the similarities between biological and artificial neuron their composition and inner workings. In the chapter Artificial Neural Networks we describe basic information about different, most commonly used artificial neural networks topologies. We described Feed-forward, Recurrent, Hopfield, Elman, Jordan, Long Short Term Memory, Bi-directional, Self Organizing Maps, Stochastic and Physical artificial neural networks. After describing various types of artificial neural networks architectures we describe how to make them useful by learning. We describe different learning paradigms (supervised, unsupervised and reinforcement learning) in chapter Learning. In the last chapter Usage of Artificial Neural Networks we describe how to handle artificial neural networks in order to make them capable of solving certain problems. In order to show what artificial neural networks are capable of, we gave a short example how to use bi-directional artificial neural network in mobile-phone fraud detection system.

7. References
Gurney, K. (1997). An Introduction to Neural Networks, Routledge, ISBN 1-85728-673-1 London Krenker A.; Volk M.; Sedlar U.; Bešter J.; Kos A. (2009). Bidirectional artificial neural networks for mobile-phone fraud detection. ETRI Jurnal., vol. 31, no. 1, Feb. 2009, pp. 92-94, COBISS.SI-ID 6951764 Kröse B.; Smagt P. (1996). An Introduction to Neural Networks, The University of Amsterdam, Amsterdam. Paveši N. (2000). Razpoznavanje vzorcev: uvod v analizo in razumevanje vidnih in slušnih signalov, Fakulteta za elektrotehniko, ISBN 961-6210-81-5, Ljubljana Rojas R. (1996). Neural Networks: A Systematic Introduction, Springer, ISBN 3-540-60505-3, Germany.

18

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Wakuya H.; Shida K.. (2001). Bi-directionalization of neural computing architecture for time series prediction. III. Application to laser intensity time record “Data Set A”. Proceedings of International Joint Conference on Neural Networks, pp. 2098 – 2103, ISBN 0-7803-7044-9, Washington DC, 2001, Washington DC.

0 1 2
Review of Input Variable Selection Methods for Review of Input Variable Selection Methods Artiﬁcial Neural Networks for Artificial Neural Networks
Robert May1 , Graeme Dandy2 and Holger Maier3
1 Veolia

Water, University of Adelaide 2,3 University of Adelaide Australia

1. Introduction
The choice of input variables is a fundamental, and yet crucial consideration in identifying the optimal functional form of statistical models. The task of selecting input variables is common to the development of all statistical models, and is largely dependent on the discovery of relationships within the available data to identify suitable predictors of the model output. In the case of parametric, or semi-parametric empirical models, the difﬁculty of the input variable selection task is somewhat alleviated by the a priori assumption of the functional form of the model, which is based on some physical interpretation of the underlying system or process being modelled. However, in the case of artiﬁcial neural networks (ANNs), and other similarly data-driven statistical modelling approaches, there is no such assumption made regarding the structure of the model. Instead, the input variables are selected from the available data, and the model is developed subsequently. The difﬁculty of selecting input variables arises due to (i) the number of available variables, which may be very large; (ii) correlations between potential input variables, which creates redundancy; and (iii) variables that have little or no predictive power. Variable subset selection has been a longstanding issue in ﬁelds of applied statistics dealing with inference and linear regression (Miller, 1984), and the advent of ANN models has only served to create new challenges in this ﬁeld. The non-linearity, inherent complexity and non-parametric nature of ANN regression make it difﬁcult to apply many existing analytical variable selection methods. The difﬁculty of selecting input variables is further exacerbated during ANN development, since the task of selecting inputs is often delegated to the ANN during the learning phase of development. A popular notion is that an ANN is adequately capable of identifying redundant and noise variables during training, and that the trained network will use only the salient input variables. ANN architectures can be built with arbitrary ﬂexibility and can be successfully trained using any combination of input variables (assuming they are good predictors). Consequently, allowances are often made for a large number of input variables, with the belief that the ability to incorporate such ﬂexibility and redundancy creates a more robust model. Such pragmatism is perhaps symptomatic of the popularisation of ANN models through machine learning, rather than statistical learning theory. ANN models are too often developed without due consideration given to the effect that the choice of input variables has on model complexity, learning difﬁculty, and performance of the subsequently trained ANN.

20

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Recently, ANN modellers have become increasingly aware of the need to undertake input variable selection (IVS), and a myriad of methods employed to undertake the IVS task are described within reported ANN applications—some more suited to ANN development than others. This review therefore serves to provide some guidance to ANN modellers, by highlighting some of the key issues surrounding variable selection within the context of ANN development, and survey some the alternative strategies that can be adopted within a general framework, and provide some examples with discussion on the beneﬁts and disadvantges in each case.

2. The input variable selection problem
Recall that for an unknown, steady-state input-output process, the development of an ANN provides the non-linear transfer function Y = F ( X ) + ε, (1)

where the model output Y is some variable of interest, X is a k-dimensional input vector, whose component variables are denoted by Xi (i = 1, . . . , k), and ε is some small random noise. Let C denote the set of d variables that are available to construct the ANN model. The Id−k problem of input variable selection (IVS) is to choose a set of k variables from C to form X (Battiti, 1994; Kwak & Choi, 2002) that leads to the optimal form of the model, F. Dynamic processes will require the development of an ANN to provide a time-series model of the general form Y (t + k) = F (Y (t), . . . , Y (t − p), X (t), . . . , X (t − p)) + ε(t). (2)

Here, the output variable is predicted at some future time t + k, as a function of past values of both input X and output Y. Past observations of each variable are referred to as lags, and the model order p deﬁnes the maximum lag of the model. The model order reﬂects the persistence of dynamics within the system. In comparison to the steady-state model formulation, the number of variables in the candidate set C is now multiplied by the model order. Consequently, for systems with strong persistence, the number of candidate variables is often quite large. ANN models may be speciﬁed with insufﬁcient, or uninformative input variables (under-speciﬁed); or more inputs than is strictly necessary (over-speciﬁed), due to the inclusion of superﬂuous variables that are uninformative, weakly informative, or redundant. Deﬁning what constitutes an optimal set of ANN input variables ﬁrst requires some consideration of the impact that the choice of input variables has on model performance. The following arguments summarise the key considerations: Relevance. Arguably the most obvious concern is that too few variables are selected, or that the selected set of input variables is not sufﬁciently informative. In this case, the outcome is a poorly performing model, since some of the behaviour of the output remains unexplained by the selected input variables. In most cases, it is reasonable to assume that a modeller will have some expert knowledge of the system under consideration; will have surveyed the available data, and will have arrived at a reasonable set of candidate input variables. The a priori assumption of model development is that at least one or more of the available candidate variables is capable of describing some, if not all, of the output behaviour, and that it is the nature and relative strength of these relationships that is unknown (which is, of course, the motivation behind the development of non-parametric

Review of Input Variable Selection Methods for Artificial Neural Networks

21

models). Should it happen that none of the available candidates are good predictors, then the problem of model development is intractable, and it may be necessary to reconsider the available data and the choice of model output, and to undertake further measurements or observations before revisiting the task of model development. Computational Effort. The immediately obvious effect of including a greater number of input variables is that the size of an ANN increases, which increases the computational burden associated with querying the network—a signiﬁcant inﬂuence in determining the speed of training. In the case of the multilayer perceptron (MLP), the input layer will have an increased number of incoming connection weights. In the case of kernel-based generalised regression neural network (GRNN) and radial basis function (RBF) networks, the computation of distance to prototype vectors is more expensive due to higher dimensionality. Furthermore, additional variables place an increased burden on any data pre-processing steps that may be undertaken during ANN development. Training difﬁculty. The task of training an ANN becomes more difﬁcult due to the inclusion of redundant and irrelevant input variables. The effect of redundant variables is to increase the number of local optima in the error function that is projected over the parameter space of the model, since there are more combinations of parameters that can yield locally optimal error values. Algorithms such as the back-propagation algorithm, which are based on gradient descent, are therefore more likely to converge to a local optimum resulting in poor generalisation performance. Training of the network is also slower because the relationship between redundant parameters and the error is more difﬁcult to map. Irrelevant variables add noise into the model, which also hinders the learning process. The training algorithm may expend resources adjusting weights that have no bearing on the output variable, or the noise may mask the important input-output relationships. Consequently, many more iterations of the training algorithm may be required to determine a near-global optimum error, which adds to the computational burden of model development. Dimensionality. The so-called curse of dimensionality (Bellman, 1961) is that, as the dimensionality of a model increases linearly, the total volume of the modelling problem domain increases exponentially. Hence, in order to map a given function over the model parameter space with sufﬁcient conﬁdence, an exponentially increasing number of samples is required (Scott, 1992). Alternatively, where a ﬁnite number of data are available (as is generally the case in real-world applications), it can be said that the conﬁdence or certainty that the true mapping has been found will diminish. ANN architectures like the MLP are particularly susceptible to the curse due to the rapid growth in the number of connection weights as input variables are added. Table 1 illustrates the growth in the sample size required to maintain a constant error associated with estimates of the input probability, as determined by the pattern layer of a GRNN. Some ANN architectures can also circumvent the curse of dimensionality through their handling of redundancy and their ability to simply ignore irrelevant variables (Sarle, 1997). Others, such as RBF networks and GRNN architectures, are unable to achieve this without signiﬁcant modiﬁcations to the behaviour of their kernel functions, and are particularly sensitive to increasing dimensionality (Specht, 1991). Comprehensibility. In many applications, such as in the case of ANN transfer functions for process modelling, it will often sufﬁce to regard an ANN as a “black-box"’ model. However, ANN modellers are increasingly concerned with the development of ANN

22

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Dimension, d Sample size, N 1 2 3 4 5 6 7 8 9 10 4 19 67 223 768 2790 10 700 43 700 180 700 842 000

Table 1. Growth of sample size with increasing dimensionality required to maintain a constant standard error of the probability of an input estimated in the GRNN pattern layer (Silverman, 1986). models for knowledge discovery from data (KDD) and data mining (Craven & Shavlik, 1998). The goal of KDD is to train an ANN based on observations of a process, and then interrogate the ANN to gain further understanding of the process behaviour it has learned. Rule-extraction from ANN models can be useful for a number of purposes, including: (i) deﬁning input domains that produce certain ANN outputs, which can be useful knowledge in itself; (ii) validation of the ANN behaviour (e.g. verifying that input-output response trends make sense), which increases conﬁdence in the ANN predictions; and (iii) the discovery of new relationships, which reveals previously unknown insights into the underlying physical process (Craven & Shavlik, 1998; Darbari, 2000). Reducing the complexity of the ANN architecture, by minimising redundancy and the size of the network, can signiﬁcantly improve the performance of data mining and rule extraction algorithms. Based on the arguments presented, a desirable input variable is a highly informative explanatory variable (i.e a good predictor) that is dissimilar to other input variables (i.e. independent). Consequently, the optimal input variable set will contain the fewest input variables required to describe the behaviour of the output variable, with a minimum degree of redundancy and with no uninformative (noise) variables. Identiﬁcation of an optimal set of input variables will lead to a more accurate, efﬁcient, cost-effective and more easily interpretible ANN model. The fundamental importance of the IVS issue is evident from the depth of literature surrounding the development and discussion of IVS algorithms in ﬁelds such as classiﬁcation, machine learning, statistical learning theory, and many other ﬁelds where ANN models are applied. In a broad context, reviews of IVS approaches have been presented by Kohavi & John (1997), Blum & Langley (1997) and more recently, by Guyon & Elisseeff (2003). However, in many examples of the application of ANNs to modelling and data analysis applications, the importance of IVS is often understated. In other cases, the task is given only marginal consideration and this often results in the application of ad hoc or inappropriate methods. Reviews by Maier & Dandy (2000) and Bowden (2003) examined the IVS methods that have been applied to ANN applications in engineering and concluded that there was a need for a more considered approach to the IVS task. Certainly, no consensus has been reached regarding

Review of Input Variable Selection Methods for Artificial Neural Networks

23

suitable methods for undertaking the IVS task in the development of ANN regression or time-series forecasting models (Bowden, 2003).

3. Taxonomy of algorithms
Figure 1 presents a taxonomy, which provides some examples of the various approaches that have been proposed within ANN literature. IVS algorithms can be broadly classiﬁed into three main classes: wrapper, embedded or ﬁlter algorithms (Blum & Langley, 1997; Guyon & Elisseeff, 2003; Kohavi & John, 1997), as shown in Figure 1. These different conceptual approaches to IVS algorithm design are illustrated in Figure 2. Wrapper algorithms, as shown in Figure 2(a), approach the IVS task as part of the optimisation of model architecture. The optimisation searches through the set, or a subset, of all possible combinations of input variables, and selects the set that yields the optimal generalisation performance of the trained ANN. As the name indicates, embedded algorithms (Figure 2(b)) for IVS are directly incorporated into the ANN training algorithm, such that the adjustment of input weights considers the impact of each input on the performance of the model, with irrelevant and/or redundant weights progressively removed as training proceeds. In contrast, IVS ﬁlters (Figure 2(c)) distinctly separate the IVS task from ANN training and instead adopt an auxiliary statistical analysis technique to measure the relevance of individual, or combinations of, input variables. Given the general basis for the formulation of both IVS wrapper and ﬁlter designs, the diversity of implementations that can possibly be conceived is immediately apparent. However, designs for wrappers and ﬁlters share the same overall components, in that, in addition to a measure of the informativeness of input variables, each class of selection algorithms requires: 1. a criterion or test to determine the inﬂuence of the selected input variable(s), and 2. a strategy for searching among the combinations of candidate input variables.
3.1 Optimality Criteria

The optimality criterion deﬁnes the interpretation of the arguments presented in Section 2 into an expression for the optimal size k and composition of the input vector, X. Optimality criteria for wrapper selection algorithms are derived from, or are exactly the same as, criteria that are ultimately used to assess the predictive performance of the trained ANN. Essentially, the wrapper approach treats the IVS task as a model selection exercise, where each model corresponds to a unique combination of input variables. Recall that the most commonly adopted measure of predictive performance for ANNs is the mean squared error (MSE), which is given by MSE = 1 n ˆ y − yj n j∑ j =1
2

(3)

ˆ where y j and y j are the actual and predicted outputs, which correspond to a set of test data. Following the development of m models, a simple strategy is to select the model that corresponds to the minimum MSE. However, the drawback of this criterion is that the “best” performing model, in terms of the MSE, is not necessarily the “optimal” model, since models with a large number of input variables tend to be biased as a result of over-ﬁtting. Consequently, it is more common to adopt an optimality criterion such as Mallows’ C p (Mallows, 1973), or the Akaike information criterion (AIC) (Akaike, 1974), which penalise overﬁtting. Both Mallows’ C p and the AIC determine the optimal number of input variables

24

Artificial Neural Networks - Methodological Advances and Biomedical Applications
Dimension Reduction Rotation Linear Principal component analysis (PCA) Partial Least-Squares (PLS) (Wold, 1966) Non-Linear Independent component analysis (ICA) Non-linear PCA (NLPCA) Clustering Learning vector quantisation (LVQ) Self-organizing map (SOM) (Bowden et al., 2002) Variable selection Model-based Wrapper Nested Forward selection (constructive ANNs) Backward elimination Nested subset (e.g. increasing delay order) Global search Exhaustive search Heuristic search (e.g. GA-ANN) Ranking Single-variable Ranking (SVR) GRNN Input Determination Algorithm (GRIDA) Embedded Optimisation Direct Optimisation (e.g. Lasso) Evolutionary ANNs Weight-based Stepwise regression Pruning (e.g. OBD (Le Cun et al., 1990)) Recursive feature elimination Filter (model-free) Correlation (linear) Rank (maximum) Pearson correlation Ranked (maximum) Spearman correlation Forward partial correlation selection Time-series analysis (Box & Jenkins, 1976) Information theoretic (non-linear) Entropy Entropy (minimum) ranking Minimum entropy Mutual Information (MI) Rank (maximum) MI MI feature selection (MIFS) (Battiti, 1994) MI w/ICA (ICAIVS) (Back & Trappenberg, 2001) Partial mutual information (PMI) (Sharma, 2000) Joint MI (JMI) (Bonnlander & Weigend, 1994)

Fig. 1. Taxonomy of IVS Strategies and Algorithms

Review of Input Variable Selection Methods for Artificial Neural Networks

25

Optimality Test

Candidates

Search Algorithm

Variable(s)

Training

Error

Model Selection

Selected ANN (inputs)

(a) Wrapper

Training

Optimality Test

Candidates

Query

Error

Weight Update

Trained ANN (inputs)

(b) Embedded

Optimality Test

Candidates

Search Algorithm

Variable(s)

Statistical Evaluation

Selected input(s)

Training

Trained ANN

(c) Filter

Fig. 2. Conceptual IVS approach using a (a) wrapper, (c) embedded, or (b) ﬁlter algorithm. by deﬁning the optimal trade-off between model size and accuracy by penalising models with an increasing number of parameters. In fact, the C p criterion is considered to be a special case of the AIC. Mallows’ C p is is deﬁned as Cp = ˆ ∑n=1 y j − y j (k) j
2 σd 2

− n + 2p,

(4)

2 where y j (k) are the outputs generated by a model using p parameters, and σd are residuals for a full model trained using all d possible input variables. C p measures the relative bias and variance of a model with p variables. The theoretical value of C p for an unbiased (optimal) model will be p, and in model selection, the model with the C p value that is closest to p is selected.

26

Artificial Neural Networks - Methodological Advances and Biomedical Applications

AIC(p)

AIC(p)
Optimal number of parameters

2p+1

-log(MSE(p))

p

Fig. 3. The Akaike Information Criterion determines the optimum trade-off between model error and size The AIC is deﬁned as ˆ ∑n=1 y j − y j (k) j
2

+ 2( p + 1). (5) n Here, the accuracy is determined by the log-likelihood, which is a function of the MSE. The complexity of the model is determined by the term p + 1, where p is the number of model parameters. Typically, the regression error decreases with increasing p, but since the model is more likely to be over-ﬁt for a ﬁxed sample size, the increasing complexity is penalised. At some point an optimal AIC is determined, which represents the optimal trade-off between model accuracy and model complexity. The optimum model is determined by minimising the AIC with respect to the number of model parameters, p. This is illustrated in Figure 3. Other model selection criteria have also been similarly derived, such as the Bayesian information criterion (BIC) (Schwarz, 1978), which is similar to the AIC, although it applies a more severe penalty of (k ln n) to the number of model parameters. The expression for the AIC in (5) assumes a linear regression model, but can be extended to non-linear regression. However, it should be noted that in this case, p + 1 no longer sufﬁciently describes the complexity of the model and other measures are required. Such measures include the effective number of parameters, or Vapnik-Chernovenkis dimension. The values of these measures are a function of the class of regression model that is estimated and the training data. The effective number of parameters, d can be determined by trace(S), where S is a matrix deﬁned by the expression
ˆ y = Sy. K T K, (6)

AIC = −n log

where the elements of K correspond For kernel regression, the hat matrix, S, is equal to to each K j ( x, h), and the complexity is therefore given by trace(K T K). Factors affecting

Review of Input Variable Selection Methods for Artificial Neural Networks

27

complexity include the number of data, the dimension of the data, and the number of basis functions. The VC-dimension is similarly deﬁned as the number of data points that can be shattered by the model (i.e. how many points in space can be uniquely separated by the regression function). However, calculating the VC-dimension of complex regression functions can be difﬁcult (Hastie et al., 2001). For MLP architectures, the VC-dimension is related to the number of connection weights, and for RBF networks the VC-dimension depends on the number of basis functions and their respective bandwidths, if different value are used for each basis function. Both the effective number of parameters and the VC-dimension revert to the value of p + 1 for linear models. In ﬁlter algorithm designs, the optimality criterion is embedded in the statistical analysis of candidate variables, which deﬁnes the interpretation of “good” input variables. In general, selection ﬁlters search amongst the candidate variables and identify suitable input variables according to the following criteria: 1. maximum relevance (MR), 2. minimum redundancy (mR), and 3. minimum redundancy–maximum Relevance (mRMR). The criterion of maximum relevance ensures that the selected input variables are highly informative by searching for variables that have a high degree of correlation with the output variable. Input ranking schemes are a prime example of MR techniques, in which the relevance is determined for each input variable with the output variable. Greedy selection can be applied to select the k most relevant variables, or a threshold value can be applied to select inputs that are relevant, and reject those which are not. The issue with MR criteria is that the selection of the k most relevant candidate variables does not strictly yield an optimal ANN. Here, Kohavi & John (1997) make the distinction between relevance and usefulness by observing that redundancy between variables can render highly relevant variables useless as predictors. Consequently, a criterion of minimum redundancy aims to ﬁnd inputs that are maximally dissimilar from one another, in order to select the most useful set of relevant variables. The application of an additional mR criterion with the existing MR criterion leads to mRMR selection criteria, where input variables are evaluated with the dual consideration of relevance, with respect to the output variable; and independence (dissimilarity), with respect to the other candidate variables (Ding & Peng, 2005). Embedded IVS considers regularisation (reducing the size or number) of the weights of a regression to minimise the complexity, while maintaining predictive performance. This involves the formulation of a training algorithm that simultaneously ﬁnds the minimum model error and model complexity, somewhat analogous to ﬁnding the optimum the AIC. If the model architecture is linear-in-the-parameters, the resulting expression can be solved directly. Depending on the model complexity term, this approach gives rise to various embedded selection algorithms, such as the Lasso (Tibrishani, 1996). However, the non-linear and non-parametric nature of ANN regression does not lend itself to this approach (Guyon & Elisseeff, 2003; Tikka, 2008). Instead, embedded selection is typically applied in ANN model development in the form of a pruning strategy, where the connection weights of a network are assessed, and insigniﬁcant weights are removed from the network. Pruning algorithms were originally developed to address the computational burden associated with training fully connected networks, given that many of the weights may be only marginally important due to redundancy within the ANN architecture. However, the strategy also offers the means of selectively removing inputs, since an input variable is eliminated by eliminating all connection

28

Artificial Neural Networks - Methodological Advances and Biomedical Applications

weights between an input and the ﬁrst hidden layer (Tikka, 2008). A criterion is required to identify which connection weights should be pruned, and several different approaches can be used to determine how weights are removed (Guyon & Elisseeff, 2003): 1. Analysis of sensitivity of training error to elimination of weights, or 2. Elimination of variables based on weight magnitude. Where the ﬁrst approach has been used, different expressions for the sensitivity of the error to the weights have led to various different algorithms. The use of derivatives of error functions or transfer functions at hidden nodes with respect to the weights are common strategies, and lead to examples of pruning algorithms such as the optimal brain damage (OBD) algorithm (Le Cun et al., 1990).
3.2 Search strategies

Search strategies applied to IVS algorithms seek to provide an efﬁcient method for searching through the many possible combinations of input variables and determining an optimal, or near optimal set, while working within computational constraints. Searches may be global, and consider many combinations; or local methods, which begin at a start location and move through the search space incrementally. The latter are also commonly referred to as nested subset techniques, since the region they explore comprises overlapping (i.e. nested) sets by incrementally adding variables. Exhaustive search simply evaluates all of the possible combinations of input variables and selects the best set according to a predetermined optimality criteria. The method is the only selection technique that is guaranteed to determine the optimal set of input variables for a given ANN model (Bonnlander & Weigend, 1994). Given the combinatorial nature of the IVS problem, the number of possible subsets that form the search space is equal to 2d , with subsets ranging in size from single input variables, to the set of all available input variables. Exhaustive evaluation of all of these possible combinations may be feasible when the dimensionality of the candidate set is low, but quickly becomes infeasible as dimensionality increases.
3.2.2 Forward selection 3.2.1 Exhaustive search

Forward selection is a linear incremental search strategy that selects individual candidate variables one at a time. In the case of wrappers, the method starts by training d single-variable ANN models and selecting the input variable that maximises the model performance-based optimality criterion. Selection then continues by iteratively training d − 1 bivariate ANN models, in each case adding a remaining candidate to the previously selected input variable. Selection is terminated when the addition of another input variable fails to improve the performance of the ANN model. In ﬁlter designs, the single most relevant candidate variable is selected ﬁrst, and then forward selection proceeds by iteratively identifying the next most relevant candidate and evaluating whether the variable should be selected, until the optimality criterion is satisﬁed. The approach is computationally efﬁcient overall, and tends to result in the selection of relatively small input variable sets, since it considers the smallest possible models, and trials increasingly larger input variable sets until the optimal set is reached. However, because forward selection does not consider all of the possible combinations, and only searches a

Review of Input Variable Selection Methods for Artificial Neural Networks

29

small subset, it is possible that the algorithm may encounter a locally optimum set of input variables and terminate prematurely. Also, due to the incremental nature of the forward search, the algorithm may ignore highly informative combinations of input variables that are only marginally relevant individually (Guyon & Elisseeff, 2003). Forward selection is said to have ﬁdelity, in that once an input variable is selected, the selection can not be undone. Step-wise selection is an extension of the forward selection approach, where input variables may also be removed at any subsequent iteration. The formulation of the step-wise approach is aimed at handling redundancy between candidate variables. For example, a variable Xa may be selected initially due to high relevance, but is later found to be inferior to the combination of two other variables, Xb and Xc , which only arises at a subsequent iteration. The initially selected input variable Xa is now redundant, and can be removed in favour of the pair Xb and Xc . A common example of this approach is step-wise regression, which is widely used for the development of linear regression models. In this wrapper approach, linear models are iteratively constructed by adding an input variable to the model, and re-estimating the model coefﬁcients. Input variables are retained based on analysis of the coefﬁcients of the newly developed model. The selection process continues until the model satisﬁes some optimality criterion, such as the AIC (see Section 3.1), that is, when k + 1 input variables are no better than the preceding k variables. Backward elimination is essentially the reverse of the forward selection approach. In this case, all d input variables are initially selected, and then the most unimportant variables are eliminated one-by-one. In wrapper selection strategies, the relative importance of an input variable may be determined by removing an input variable Xi and evaluating the effect on the model that is retrained without it; or, by examining the inﬂuence of each of the input variables on the output y through some sensitivity analysis. In ﬁlter strategies, the least relevant candidates are iteratively removed until the optimality criterion is satisﬁed. In general, backward elimination is inefﬁcient in comparison with forward selection, as it can require the development and evaluation of many large ANN models before reaching the optimal model. Since all input variables are initially included, it may be more difﬁcult to determine the relative importance of an individual input variable than in forward selection, which starts with a single input variable. Also, wrapper algorithms based on backward elimination may potentially be biased by overﬁtting of large models.
3.2.5 Heuristic search 3.2.4 Backward elimination 3.2.3 Step-wise selection

Heuristic search techniques are widely used in optimisation problems where the search space is large. Heuristic search algorithms are particularly adept at efﬁciently ﬁnding global, or near-global optimum solutions within large search spaces by exploiting the common attributes of good solutions. In general, the various algorithms each implement a search that combines random evaluation of solutions throughout the entire search space, with a mechanism to increase the focus of the search in regions that lead to good solutions. Examples of heuristic search algorithms applied to IVS include evolutionary algorithms (EAs), such as genetic algorithms (GAs) (Bowden, 2003) and ant colony optimization (ACO) (Izrailev & Agraﬁotis, 2002; Marcoulides & Drezner, 2003; Shen et al., 2005).

30

Artificial Neural Networks - Methodological Advances and Biomedical Applications

The application of heuristic optimisation techniques to IVS overcomes the signiﬁcant computational requirement of exhaustive search, while maintaining the desirable characteristic of providing a global (or, near-global) optimum. Moreover, EA-based IVS wrappers are an attractive option because they can also be included as part of evolutionary ANN training algorithms, which also seek to determine optimal ANN parameter values by minimising the ANN cross-validation error. However, the application of heuristic search techniques requires calibration of search algorithm parameters, which is itself not a trivial task. In general, setting the search parameters involves a trade-off between the amount the search space that is explored, and the rate at which the algorithm converges to a ﬁnal solution. Finally, heuristic algorithms retain a certain degree of randomness, and although they search more solutions in comparison to sequential selection algorithms, there is still no guarantee that the sub-space explored will include the globally optimal solution.

4. Dimensionality reduction
The taxonomical classiﬁcation in Figure 1 includes dimensionality reduction algorithms as a class of algorithms reducing the number of variables within a dataset. Dimensionality reduction is often performed in order to reduce the computational effort associated with data processing, or to identify a suitable subset of variables to include in the analysis. Although the goal of dimension reduction differs from that of variable selection, it is a closely related area and is a regularly employed data pre-processing step in many multivariate data analysis applications, and it is worth including some discussion since many dimensionality reduction techniques are employed for IVS. The following considers some common examples of dimension reduction algorithms. Comprehensive surveys of dimensionality reduction techniques can be found in Carreira-Perpinan (1997) and Fodor (2002). Principal component analysis (PCA) is a commonly adopted technique for reducing the dimensionality of a dataset X. PCA achieves dimensionality reduction by expressing the p variables x1 , . . . , x p as d feature vectors (or, principal components (PCs)), where d < p. The PCs are a set of orthogonal, linear combinations of the original variables within the dataset. Essentially, PCA can be considered a data pre-processing algorithm that determines an optimal rotational transformation of the dataset, X, that maximises the amount of variance of the output Y that is explained by the PCs (Fodor, 2002). Considering a given dataset X, PCA is performed as follows: ii. Find the covariance matrix Σ = Cov( X ) = X T X. iii. Determine the unit eigenvectors e1 , . . . , e p of Σ. iv. Determine the corresponding eigenvalues λ1 , . . . , λ p . v. Rank the eigenvectors according to their eigenvalues. vi. Select the d PCs according to their eigenvalues. Selection of PCs is based on examining the eigenvalues of each PC, which correspond to the amount of variance explained by each PC, and thereby including only the signiﬁcant PCs as input features. A common selection method is to rank the PCs and select all PCs whose eigenvalues exceed some threshold λ0 , or generate a plot of the cumulative eigenvalue as a function of the number of PCs, k, to ensure the selected components explain the desired ¯ i. Subtract the mean value of each variable, to ensure that xi = 0 for each xi ∈ X.
4.1 Principal component analysis

Review of Input Variable Selection Methods for Artificial Neural Networks

31

amount of variance of Y. Another technique is to use and generate a scree plot of the percentage contribution of each kt h PC and to visually identify an optimal value of k (Fodor, 2002). PCA has been used as the basis for IVS for the development of ANN models (see, for example, Olsson et al. (2004), Gibbs et al. (2006), and Bowden (2003)). However, the mixing of input variables is assumed to be linear, as is the relationship between principal components and the output. Consequently, the application of PCA in this case is ﬂawed, since it will fail to identify any non-linear relationships within the data. Although non-linear versions of the PCA algorithm exist, the transformations of the data can be highly complex, and interpretation of the PCs is much more difﬁcult. An additional disadvantage of PCA is that the algorithm identiﬁes important component vectors, rather than variables. Consequently, although PCA may be useful in removing noise from the data, it is not possible to distinguish the unique contributions of individual variables to the variance in the output. Independent component analysis (ICA) seeks to determine a set of d independent component vectors within a dataset X. The approach is conceptually similar to PCA, although it relaxes the orthogonality constraint on component vectors. Furthermore, where PCA determines the optimal transformation of the data by considering covariance and identifying uncorrelated PCs based on covariance, ICA considers statistically independent combinations of variables where the order of the statistic that is used can be arbitrary (Fodor, 2002). ICA is therefore not restricted to linear correlations, and is more widely applicable to non-linear datasets (Back & Trappenberg, 2001). However, like PCA, ICA cannot discriminate unique variables as predictors, and is restricted to determining independent feature vectors. Vector quantization (VQ) refers to techniques that describe a larger set of n vectors by c codebook, or prototype vectors. VQ is closely associated with data clustering and is more commonly associated with algorithms for data compression, in terms of length n. Bowden (2003) demonstrates the potential for the self-organising map (SOM) to perform vector quantisation as an alternative to PCA for data dimensionality reduction. In this case, the d vectors of the candidate set are represented by the prototype vectors of the SOM. Similar candidate variables will be identiﬁed by the formation of groups, which have the closest proximity (deﬁned by some distance measure) to the same prototype vector. However, the distance metric needs to be carefully selected, since the Euclidean distance used in SOM applications is not strictly a measure of correlation. Using linear correlation or covariance will cluster based on the strength of linear dependence between variables. Other measures, such as entropy or mutual information (MI), would be more suitable for clustering ANN variables, since they will measure non-linear dependence.
4.3 Vector quantization 4.2 Independent component analysis

5. Wrappers
Wrapper algorithms are the ﬁrst of the three main classes of variable selection algorithm shown according to Figure 1. Wrapper algorithms are the simplest IVS algorithm to formulate. Essentially, the algorithm that results is deﬁned by the choice of the induction algorithm (i.e. model architecture). The efﬁciency of a wrapper algorithm will depend on the ability of the model to represent relationships within the data; and how efﬁciently trial models can be constructed and evaluated.

32

Artificial Neural Networks - Methodological Advances and Biomedical Applications

5.1 Single variable regression (SVR)

The notion of ranking individual candidate variables according to correlation can be extended by implementing a wrapper approach in order to relax the assumption of linearity in correlation analysis (Guyon & Elisseeff, 2003). In this approach, a single variable regression1 (SVR) is constructed using each candidate variable, which is then ranked according to the model performance-based optimality criterion, such as the cross-validation error. In comparison to ranking ﬁlters, SVR can potentially suffer from overﬁtting due to the additional ﬂexibility in the regression model. The GRNN input determination algorithm (GRIDA) (Bowden et al., 2006) is a recent example of an SVR wrapper for input variable ranking, which proceeds as follows: ii. For each x ∈ X, i. Let X → C. (Initialisation) Train a GRNN and determine MSEx . For b = 1 to 100, (Bootstrap) Estimate MSEε,b . Estimate MSEε If MSEx >
(95)

iii.

iv. v. vi. vii. viii. ix.

Randomly shufﬂe x → ε. . or MSEx > Θ (Selection),

(95) MSEε

Remove x from X.
(95)

x. Return X. where MSEε is the 95th percentile, and Θ is some threshold value. Considering each variable in turn, a GRNN is trained, and then the MSE of the model is determined for a set of test data. However, rather than greedy selection of the k best variables, each variable is compared to a bootstrap estimate of a conﬁdence bound for the randomised model error, MSEε . A variable is rejected immediately if the model error exceeds the randomised error, since it is no better predictor than a random noise variable. Further strictness on selections is imposed through the heuristic error threshold, Θ. However, a suitable value for Θ needs to be determined ﬁrst. The number of variables selected for a given value of Θ will be dependent on several factors, including the degree of noise in the data, the error function used, and the distribution of the error over the candidate variables. Consequently, optimal values for Θ can only be determined for each dataset by trial and error. The estimation of the conﬁdence bound on the error for each SVR is a signiﬁcant computational requirement. Given the assumed constraint 0 < Θ < MSEε , the estimation of the bootstrap may not even be necessary to perform IVS. However, the method does provide useful information in discriminating noise variables from weakly informative ones. SVR does not consider interractions between variables, and may select redundant variables. In order to overcome this, dimensionality reduction is required as a pre-processing step, in order to obtain an independent set of candidate variables, prior to variable selection. Such an approach was used in the example of the GRNN-based SVR IVS algorithm (Bowden et al., 2006), where a SOM was used to achieve dimension reduction.
1

(95)

(95)

The term has been adapted from the term single variable classiﬁer (SVC), which is more often referred to within literature due to its application in classiﬁcation

Review of Input Variable Selection Methods for Artificial Neural Networks

33

5.2 GA-ANN

Heuristic search techniques are ideally suited to IVS wrapper designs, since they provide an efﬁcient search of the combinations of candidate input variables, with the simplicity of the black-box wrapper approach. Bowden et al. (2005) utilised an evolutionary wrapper strategy for IVS that combined a genetic algorithm (GA) optimisation with a generalised regression neural network (GRNN). The method exploits the fast GRNN training times, and the ﬁxed architecture of the GRNN, which avoids the need to optimise the internal architecture and training algorithm parameters. These are required for the development of other architectures, such as the MLP. A simple binary GA (a GA with decisions encoded as 1 or 0 within a binary string) was utilised, with the objective of minimising the MSE obtained by hold-out validation on a set of test data. In order to overcome the inability of the wrapper methodology to detect interractions between candidate variables, as with GRIDA, Bowden et al. (2005) adopted SOM-based dimensionality reduction as a pre-processing stage to reduce the candidate variables to a subset of independent variables.

6. Embedded algorithms
Embedded algorithms are a distinct class of IVS algorithm where the selection of variables is embedded within the ANN training algorithm. The distinction between wrapper and embedded algorithms is not always obvious, since embedded selection also involves iterative update and evaluation of the model parameters based on model performance. The key difference is that the evaluation of input variables occurs within the training algorithm, and only a single model is trained. Embedded algorithms also consider the impact of each individual variable on the performance of the model, while wrapper algorithms simply consider model performance for a given set of input variables as a whole.
6.1 Recursive feature elimination

Recursive feature elimination (RFE) is an implementation of backward-elimination as an embedded IVS algorithm (Guyon & Elisseeff, 2003). The RFE approach involves an iterative process of training an ANN model, initially using all candidate input variables, and then removing one or more input variables at each iteration based on the rank magnitude of the weights corresponding to each input variable. The technique has been developed and successfully applied using support vector machines (SVMs), but is extensible to other ANN architectures by providing a suitable expression for the overall connection weight for a given input variable.
6.2 Evolutionary ANNs

Evolutionary ANNs (EANNs) utilise an evolutionary algorithm to determine the optimal set of weights, where the optimisation is formulated with ANN weights as the decision variables, and an objective function incorporating model error, with penalties for model complexity. Due to the complexity of the ANN error surface that is projected over the weight-space, evolutionary algorithms have been found to be a good alternative to gradient descent algorithms. EAs are robust and able to more reliably ﬁnd a near-globally optimum solution, even for highly non-linear functions, with many multiple local optima; whereas gradient descent algorithms have greater potential to converge prematurely at a local minimum. By applying penalty terms to avoid large weights, or by using a model sparsity term within the objective function, the optimisation can effectively determine the optimum trade-off between model error and model complexity during training (Tikka, 2008).

34

Artificial Neural Networks - Methodological Advances and Biomedical Applications

IVS is embedded within the EANN approach, since the optimisation will implicitly exclude input variables by setting the input connection weight values close to, or equal to zero. Alternatively, the approach may also be made more explicit by extending the optimisation formulation to include the inclusion/exclusion of input variables within the ANN architecture as a separate binary decision variable. This approach is therefore similar to the GA-ANN wrapper approach, but extends the optimisation to the weight parameters.

7. Filters
In the taxonomy shown in Figure 1, ﬁlter algorithms represent the second sub-class of variable selection algorithms and represent an alternative to the wrapper approach. The design of ﬁlter algorithms is typically deﬁned by the measure of relevance that is used to distinguish the important input variables, as well as the optimality criteria, as they have been previously deﬁned for ﬁlters in Section 3.1. Incremental search strategies tend to dominate ﬁlter designs, since the relevance measure is usually a bivariate statistic, which necessitates evaluating each candidate-output relationship. Currently, two broad classes of ﬁlters have been considered: those based on linear correlation; and those based on information theoretic measures, such as mutual information. Arguably the most commonly used relevance measure in multivariate statistics is the Pearson correlation. The Pearson correlation (also called linear correlation, or cross-correlation), R, is deﬁned by R XY = n ¯ ¯ ∑i=1 ( xi − x )(yi − y)

7.1 Rank correlation

n n ¯ ¯ ∑ i =1 ( x i − x )2 ∑ i =1 ( y i − y )2

(7)

where R XY is the short-hand notation for R( X, Y ). In (7), the numerator is simply the sample covariance, Var ( X, Y ); and the two terms in the denominator are the square-root of the sample variances, Var ( X ) and Var (Y ). The application of correlation analysis to variable selection originates from linear regression analysis. The squared correlation, R2 , is the coefﬁcient XY of determination, and if X and Y have been standardised to have a zero mean, R2 is the equivalent to the coefﬁcient of a linear ﬁt between X and Y. Input variable ranking based on the Pearson correlation is one of the most widely used IVS methods. The selection of candidate variables that are sorted by order of decreasing correlation is based either on greedy selection of the ﬁrst k variables, or upon all variables for which the correlation is signiﬁcantly different from zero. The signiﬁcance of the Pearson correlation can be determined directly, since the error associated with estimation of correlation from a sample is deﬁned by the t-distribution. A rule of thumb (for large n) is that variables √ with an absolute correlation greater than 2/ n are signiﬁcant. Identiﬁcation of signiﬁcant correlations is a common technique in data mining applications, such as gene expression analysis, where the goal is simply to mark potentially important genes for further investigation. However, in terms of IVS algorithms, the method is classed as an MR ﬁlter, and does not consider interactions between variables. Redundancy is particularly problematic for multivariate time-series forecasting, which considers lagged values that are often highly correlated (auto-correlated).

Review of Input Variable Selection Methods for Artificial Neural Networks

35

7.2 Partial correlation

In the case where candidate variables are themselves correlated, redundancy becomes an important issue. In such cases, a correlation ranking approach is likely to select too many variables, since many candidates will each provide the same information regarding the output variable. Given three variables X, Y and Z, the partial correlation, R ( X, Y | Z ) measures the correlation between X and Y after the relationship betwen Y and Z has been discounted. The partial correlation can be determined from the Pearson correlation using the equation: R XY · Z
2 (1 − R2 )(1 − RYZ ) XZ

R XY − R XZ RYZ

(8)

where R XY · Z and R XY etc. are the short-hand notation for R ( X, Y | Z ) and R XY etc. Partial correlation is similar to stepwise multiple linear regression. The subtle difference is that in stepwise MLR, successive models are ﬁtted with additional input variables, and variables are selected (or later rejected) based on the estimated model coefﬁcients. However, in partial correlation analysis, the magnitude of R for each variable is not necessarily equal to the regression coefﬁcients for a ﬁtted MLR model, since redundancy between variables means that the solution to the MLR parameter estimation is a line (two redundant coefﬁcients) or a surface, that is, there will be inﬁnite combinations of equivalent model coefﬁcients. The partial correlations obtained are in fact one speciﬁc solution to the MLR parameter estimation. Another difference is that forward selection is used in partial correlation analysis, because once the most salient variable has been selected, it will not be rejected later, and the partial correlations of subsequent variables will be dependent on those already selected.
7.3 Box-Jenkins

Box-Jenkins time-series analysis (Box & Jenkins, 1976), which considers the development of linear auto-regressive, moving-average (ARMA) models to represent dynamic processes, is the most common approach to the development of time-series and process transfer functions. ARMA models are described by the general form y ( t + 1) =

k =0

∑ αk y(t − k ) + ∑ β k u(t − k ) k =0

p

q

(9)

where αk and β k are coefﬁcients and p and q denote the order of the autoregressive (AR) and moving-average (MA) components of the model, respectively. Identiﬁcation of the optimal model parameters p and q forms the goal of Box-Jenkins model identiﬁcation, and hence variable selection. The autocorrelation function (ACF), R (Y (t − k), Y (t)), determines q and the partial autocorrelation function (PACF) determines p. The ACF is determined for a given time-series sample by Rk = n− ¯ ¯ ∑i=1k ( xi − x )( xi−k − x ) n ¯ ( x i − x )2 ∑ i =1

(10)

where Rk is the short-hand notation for the auto-correlation of a time-series with a delay of k. The PACF at a delay of k is denoted by φkk , and is estimated from the ACF based on the following series of equations

36

Artificial Neural Networks - Methodological Advances and Biomedical Applications

φ11 = R1 φ22 = R2 − R2 1 1 − R2 1

(11) (12) (13) (14)

φkj = φk−1,j − φkk φk−1,k− j , for k ≥ 2 and j ≥ 1, φkk =
−1 Rk − ∑k=1 φk−1,j Rk− j j −1 1 − ∑k=1 φk−1,j R j j

, for k ≥ 3.

The Box-Jenkins methodology can be used to similarly identify optimal linear autoregressive with exogenous inputs (ARX) models. In this case, the partial cross-correlation is used to identify the relevant lags of the exogenous variables. Box-Jenkins and partial autocorrelation analysis have been used as the basis for IVS in the development of ANN models. In some examples, ANNs have been developed based on an optimal set determined for an ARX model. The ANNs were found to produce better predictions than the ARX model, and this has often provided the justiﬁcation for ANN modelling in favour of conventional time-series techniques (Rodriguez et al., 1997). However, although this demonstrated the additional ﬂexibility of ANN architectures to describe more complex behaviour, the ANN developed may not have been optimal, since the selection of inputs was based on the identiﬁcation of a linear model. It may be the case that variables that are highly informative, but non-linearly correlated with the output variable, will be overlooked and excluded from the ANN model.
7.4 Mutual information

The limitations of linear correlation analysis have created interest in alternative statistical measures of dependence, which are more adept at identifying and quantifying dependence that may be chaotic or non-linear; and which may therefore be more suitable for the development of ANN models. Mutual information (MI) is a measure of dependence that is based on information theory and the notion of entropy Shannon (1948), and is determined by the equation I ( X; Y ) = p( x, y) log p( x, y) dxdy, p( x ) p(y) (15)

where I denotes the MI between X and Y. MI measures the quantity of information about a variable Y that is provided by a second variable X. However, it is often convenient to simply regard MI as a more general measure of correlation, since despite originating from information theory, rather than statistics, MI is not entirely unrelated to Pearson correlation. In fact, it can be shown that in the case of noise-free, Gaussian data, MI will be related to linear correlation according the relationship: 1 log 1 − R2 (16) XY . 2 The advantage of MI over linear correlation is that MI is based solely on probability distributions within the data and is therefore an arbitrary measure, which makes no assumption regarding the structure of the dependence between variables. It has also been found to be robust due to its insensitivity to noise and data transformations (Battiti, 1994; I ( X; Y ) =

Review of Input Variable Selection Methods for Artificial Neural Networks

37

Darbellay, 1999; Sooﬁ & Retzer, 2003). Consequently, MI has recently been found to be a more suitable measure of dependence for IVS during ANN development. Torkkola (2003) also discusses the merit of analysing MI, given that it provides an approximation to the Bayes error rate. Bayes’ theorem, which gives the most general form of statistical inference, is given by p(y| x ∈ X ) = p ( y ∈ Y ) p ( x ∈ X |Y ) p( x ∈ X ) (17)

Bayes’ theorem can be used to determine the expectation E (y| x ∈ X ), assuming the probability distributions are known. MI provides an approximation to the error associated with the Bayes estimate of E(y| X ∈ X ), since it can be shown that the minimum error will be achieved for a maximal value of I ( X; Y ). In effect, MI provides a generic estimation of the modellability of an output variable Y, which therefore makes MI an attractive measure of relevance in determining an optimal set of input variables, since we would seek the set of input variables that maximises the JMI, that is, the MI between the output and the input variable set. The MIFS algorithm is a forward selection ﬁlter proposed by Battiti (1994) to address shortcomings with algorithms based on linear correlation. Considering the candidate set C and output variable Y, the MIFS algorithm proceeds as follows: ii. While | X | < k, i. Let X → φ.

iii. iv. v. vi.

For each c ∈ C,

Move cs to X.

Find cs that maximises I (c, Y | X ).

Estimate I (c, Y | X ) = I (c, Y ) − β ∑ x∈ X I (c; x ).

vii. Return X. MIFS deﬁnes a MR ﬁlter and identiﬁes suitable candidates according to the estimated bivariate MI between candidate variables and the output variable. The MI between the most salient candidate cs and the already selected variables in X is estimated and subtracted from the relevance in order to achieve minimum redundancy. The heuristic weighting β determines the degree of redundancy checking within MIFS. If β = 0, then MIFS will neglect relationships between candidates and MIFS is reduced to a MI ranking ﬁlter. Increasing β increases the inﬂuence of candidate redundancy on selections, however if β is too large, then the redundancy is overstated and candidate interactions dominate the selection of variables, rather than the input-output relationships (Kwak & Choi, 2002). Battiti (1994) recommends that a weighting of 0.5–1.0 is appropriate. A criticism of the forward selection approach is that the JMI of the input variable set must be considered in order to correctly determine the optimality of the input variables (Bonnlander & Weigend, 1994). However, in MIFS, the forward selection procedure considers variables individually, and optimality of the JMI is inferred by the mRMR selection. The heuristic redundancy parameter β provides only an approximation to the conditional dependence and does not necessarily relate to the JMI.

38

Artificial Neural Networks - Methodological Advances and Biomedical Applications

7.5 Partial mutual information

Sharma (2000) proposed an IVS ﬁlter that is structured similarly to MIFS, but is based instead upon direct estimation of partial mutual information (PMI). The algorithm has been successfully applied to select predictors for hydrological models (Sharma, 2000) and ANN water quality forecasting models (Bowden et al., 2002; Kingston, 2006). The PMI-based ﬁlter also incorporates a mechanism for testing the signiﬁcance of candidate variables, so that the termination point of the algorithm is optimally determined, which is an improvement over the greedy selection of k variables in MIFS. In this case, the termination criterion is based upon the distribution of the error in PMI estimation, which is numerically approximated by a bootstrap approach (Sharma, 2000). The signiﬁcance of the most relevant candidate is determined by direct comparison to the upper conﬁdence bound on the estimation error. The details of the algorithm, are as follows (May et al., 2009a):
1: Let X → φ (Initialisation) 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17:

While C = φ (Forward selection) ˆ Construct kernel regression estimator mY (X) ˆ Calculate residual output u = Y − mY (X) For each c ∈ C ˆ Construct kernel regression estimator mc (X) ˆ Calculate residual candidate v = c − mc (X) Estimate I (v; u) Find candidate cs (and vs ) that maximises I (v; u) For b = 1 to B (Bootstrap) Randomly shufﬂe vs to obtain v∗ s Estimate Ib = I (v∗ ; u) s Find conﬁdence bound Ib If I (vs , u) > Ib Else Break
(95) (95)

(Selection/termination)

Move cs to X

18: Return selected input set X.

Here, B is the bootstrap size; and Ib denotes the 95th percentile bootstrap estimate of the randomised PMI, Ib . The algorithm is structured in a similar fashion to MIFS (Battiti, 1994), but has two advantages. First, PMIS inherently handles redundancy within the candidate set through the direct estimation of PMI, whereas MIFS approximates the effect of selected inputs by means of a heuristic weighting factor. Second, while MIFS uses greedy selection of a pre-speciﬁed number of input variables, PMIS includes a criterion that automatically determines the optimum point at which to terminate the selection procedure. The optimality of the input variable set is ensured because PMI is directly estimated, and the JMI can be

(95)

Review of Input Variable Selection Methods for Artificial Neural Networks

39

determined as a result of the MI chain-rule decomposition, which is given as (Cover & Thomas, 1991) I ( x1 , . . . , x p ; y ) = I ( x1 ; y ) + I ( x2 ; y | x1 ) + · · · + I ( x p ; y | x1 , . . . , x p −1 ). (18)

Recall that in MIFS, the JMI cannot be directly approximated because redundancy is only approximated by a heuristic weighting factor. The termination criterion in PMIS automatically determines the optimal number of input variables, since the increase in JMI is additive, and once the contribution of an additional input variable is insigniﬁcant, the selection process terminates and the JMI will be maximised. An additional beneﬁt of the PMI-based approach is that the information yield during IVS provides a useful indication of the contribution of each input variable to the prediction of the output variable. Several methods for determining the usefulness of input variables based on analysis of the trained model have been described and range from sensitivity analysis, to aggregation of the weights associated with each input variable. However, the relative importance of an input variable can be determined statistically from the MI between each input and the output variable (Sooﬁ & Retzer, 2003). The PMI estimated for a given variable can potentially also be used to classify input variables as informative, or weakly informative, as deﬁned by Kohavi & John (1997), by considering the conditional relevance. Kingston (2006) considered several techniques for determining the relative importance (RI) of input variables and found that the method based on PMI yielded similar estimates of RI as methods based on analysis of the connection weights for a trained MLP. The method for estimating RI was based on the formula RI(i ) = I ( xi ; y ) , ∑ x∈ X I ( x; y) (19)

where I denotes the PMI estimated for candidate variable x during PMIS. The usefulness of RI is that it provides an indication of the way in which the ANN generates predictions. Although it is assumed that the ANN is using all of the input variables, it may in fact only require some small subset of the available input variables to generate predictions. In this case, further reﬁnements to the ANN can be made based on this interpretation. Such considerations might be important when considering the cost of data collection that is associated with ongoing deployment of ANN models. One might consider sacriﬁcing model accuracy in favour of cost reductions in ANN maintenance by reducing the number of input variables even further, which would reduce data requirements. The RI of variables may also encourage increased efforts toward the development of measurement techniques to ensure data quality for important variables. The main limitation of the PMI ﬁlter is the computational effort associated with the bootstrap estimation of MI. Even obtaining a single estimate of MI is an expensive computation due to the O{n2 } density estimation, and computational efﬁciency is therefore inﬂuenced by the sample size, n. The use of a bootstrap-based test for signiﬁcance further adds to the overall computational burden by increasing the number of estimations of MI required to implement IVS. Sharma (2000) restricts the size of the bootstrap to 100 in order to maintain reasonable analysis times. However, a small bootstrap of this size might compromise the accuracy of the termination criterion, since potentially the conﬁdence bounds may be poorly estimated. May et al. (2009b) introduce several alternative termination criteria that eliminate the bootstrap signiﬁcance-test from the algorithm. These were found to provide a signiﬁcant improvement in both the accuracy and computational effort associated with the PMI-based algorithm.

40

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Fernando et al. (2009) also present a fast implementation of the PMI-based ﬁlter design using an average-shifted histogram (ASH) (Silverman, 1986), rather than kernel density estimation, for the computation of PMI during selection.
7.6 ICAIVS

A hybrid ICA and IVS ﬁlter algorithm (ICAIVS) was proposed by Back & Trappenberg (2001), which consists of two main steps: (Trappenberg et al., 2006) i. Produce a set of candidates which are as independent as possible (ICA). ii. Measure relevance between the independent candidate variables and the desired output variables (IVS). Here, the statistical analysis of input relevance is based on estimation of the joint dependence p of combinations of input variables and considers all combinations from c( x1 , y) through to p p c( x1 , . . . , xn , y), where p denotes the order of the dependence that is measured. The IVS procedure then compares the relevance for each subset of variables, with respect to the average dependence for all subsets, and a subset is selected if the dependence exceeds some threshold value K. A drawback of ICAIVS is that the algorithm does not scale well, given the large number (3n − 1) of statistical tests that must be performed, considering only second-order statistics. Recently, an improved version of ICAIVS was described that utilised MI as the statistical measure of dependence (Trappenberg et al., 2006). This reduced the number of statistical tests by considering only ﬁrst order MI. However, the problem of specifying a suitable threshold value still remains. Both Back & Trappenberg (2001) and Trappenberg et al. (2006) used a value of 0.2 for this threshold. However, in this case the threshold value is heuristically determined, and a suitable value may vary depending on the dataset.

8. Summary
Input variable selection remains an important part of ANN model development, due to the negative impact that poor selection can have on the performance of ANNs during training and deployment post-development. The brief taxonomy of variable selections indicates the different classes of algorithms that have been employed to perform the IVS task, of which many more variations within each class are described within ANN literature. A summary of the attributes of different IVS algorithms is given in Table 2. Several key considerations should guide a modeller in determining the most appropriate approach to take for IVS in any given circumstance. The ﬁrst consideration is whether the algorithm is suitable for identifying non-linear relationships, which is a fundamental requirement for the development of ANN models. Many methods applied successfully within the context of linear regression are not suited to ANN development, where there is a presumption of non-linearity, and important relationships within the data may not be identiﬁed by a linear selection algorithm. The choice between model-free and model-based IVS algorithms is also a key consideration. The restriction imposed by wrapper designs on the choice of model architecture, and the computational requirements associated with training multiple model instances, will dictate which approach to use. Where portability between ANN architectures is not required, the use of wrapper or embedded algorithms is likely to be straightforward to implement, and are a pragmatic approach to take. However, in all cases a wrapper that includes a regularisation

Review of Input Variable Selection Methods for Artificial Neural Networks

41

Algorithm Rank-R Box-Jenkins R’ SVR Forward selection Backward selection GA-ANN MIFS PMIS RFE EANN

Criterion Correlation ACF/PACF Correlation Rank error Error Error Error MI PMI Weight size Error

Type Filter Filter Filter Wrapper Wrapper Wrapper Wrapper Filter Filter Embedded Embedded

Non-linear Redundancy Search No No No Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes No Yes Yes Yes Yes Yes Yes Yes Greedy Greedy Stepwise Greedy Nested Nested Heuristic Nested Nested Nested Heuristic

Table 2. Characteristics of input variable selection algorithms term, or considers the impact of model size, should be used to achieve an optimal trade-off between model performance and size. High computational requirements are the main drawback of model-based techniques. The training time for a single instance of an ANN will be a strong indication of the feasibility of a model-based approach. Popular methods for wrappers and embedded algorithms are typically applied where the number of training examples is relatively small, and the number of candidate input variables is large. Alternatively, ﬁxed ANN architectures like the GRNN provide fast-training, which increases the suitability of model-based IVS. The search strategy employed in a wrapper approach also represents a balance between the number of unique solutions considered, and the computational effort expended. Backward elimination algorithms, such as RFE, train relatively few ANN models and can be highly efﬁcient, but will restrict the search due to nesting behaviour. Brute force search is likely to be infeasible in most cases, but evolutionary search approaches can provide a suitable compromise that allows for greater coverage of combinations of input variables. If evolutionary optimisation is to be used for ANN training, than the embedded EANN approach may be appropriate, as an extension of weight optimisation. In contrast to model-based IVS, ﬁlter algorithms offer a fast, model-free approach to variable selection, and are particularly suited to applications where independence from a speciﬁc ANN architecture is required; or, where computational efﬁciency is sought. A generic estimation of input variable importance and redundancy avoids the traps of model over-ﬁtting, or idiosyncrasies of a speciﬁc ANN architecture. The ability to identify an optimal set of input variables prior to training an ANN eliminates the computational burden associated with training and model selection, which can reduce the overall effort of ANN development. Most ﬁlter designs are based on bivariate statistics, since estimation of statistics for multivariate data is often too inaccurate to facilitate IVS due to ﬁnite-sample error, and typically use a ranking scheme or sequential forward selection scheme. Simple ranking schemes are greedy and select more input variables than necessary, by ignoring redundancy of candidates, and are not ideally suited to multivariate ANN regression. Forward selection provides an efﬁcient search, and greediness can be overcome provided that adequate redundancy checking is incorporated into the statistical analysis of variables. This ensures that the approach selects the most informative input set with the smallest number of variables.

42

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Mutual information provides a generic measure of the relevance of a candidate input variable and is highly suitable for ANN development because it measures the strength of both linear and non-linear relationships. MI is also less sensitive to data transformations than correlation, which makes it more reliable, even for linear data analysis applications. The PMI-based forward selection approach provides a forward selection strategy that both minimises redundancy and maximises relevance. The approach determines the optimal set of input variables by estimating the maximum joint mutual information. Estimation of MI is more computationally intensive than estimation of correlation, due to the need to estimate density functions of the data. However, several reﬁnements related to estimation of critical values, and the use of faster methods for density estimation can signiﬁcantly reduce the computational effort of this approach.

9. References
Akaike, H. (1974). A new look at the statistical model identiﬁcation, IEEE Transactions of Automatic Control 19: 716–723. Back, A. D. & Trappenberg, T. P. (2001). Selecting inputs for modeling using normalized higher order statistics and independent component analysis, IEEE Transactions on Neural Networks 12(3): 612–617. Battiti, R. (1994). Using mutual information for selecting features in supervised neural net learning, IEEE Transactions on Neural Networks 5(4): 537–550. Bellman, R. (1961). Adaptive control processes: a guided tour, Princeton University Press, New Jersey. Blum, A. & Langley, P. (1997). Selection of relevant features and examples in machine learning, Artiﬁcial Intelligence 97(1–2): 245–271. Bonnlander, B. V. & Weigend, A. S. (1994). Selecting input variables using mutual information and nonparametric density estimation, International Symposium on Artiﬁcial Neural Networks, Taiwan, pp. 42–50. Bowden, G. J. (2003). Forecasting water resources variables using artiﬁcial neural techniques, Ph.d, University of Adelaide. Bowden, G. J., Dandy, G. C. & Maier, H. R. (2005). Input determination for neural network models in water resources applications. part 1 - background and methodology, Journal of Hydrology 301(1-4): 75–92. Bowden, G. J., Maier, H. R. & Dandy, G. C. (2002). Optimal division of data for neural network models in water resources applications, Water Resources Research 38(2): 1–11. Bowden, G. J., Nixon, J. B., Dandy, G. C., Maier, H. R. & Holmes, M. (2006). Forecasting chlorine residuals in a water distribution system using a general regression neural network, Mathematical and Computer Modelling 44(5-6): 469–484. Box, G. E. P. & Jenkins, G. M. (1976). Time Series Analysis, Forecasting and Control, Holden-Day Inc., San Francisco. Carreira-Perpinan, M. A. (1997). A review of dimension reduction techniques, Technical report, Dept. of Computer Science, University of Shefﬁeld. Cover, T. M. & Thomas, J. A. (1991). Elements of information theory, Wiley series in telecommunications, John Wiley & Sons, Inc., New York. Craven, M. W. & Shavlik, J. W. (1998). Using neural networks for data mining, Future Generation Computer Systems 13(2–3): 211–229. Darbari, A. (2000). Rule extraction from trained ANN: A survey, Technical report, Institute of Artiﬁcial Intelligence, Dept. of Computer Science, TU Dresden.

Review of Input Variable Selection Methods for Artificial Neural Networks

43

Darbellay, G. A. (1999). An estimator of the mutual information based on a criterion for independence, Computational Statistics & Data Analysis 32: 1–17. Ding, C. & Peng, H. (2005). Minimum redundancy feature selection from microarray gene expression data, Journal of Bioinformatics and Computational Biology 3(2): 185–205. Fernando, T. M. K. G., Maier, H. R. & Dandy, G. C. (2009). Selection of input variables for data driven models: An average shifted histogram partial mutual information estimator approach., Journal of Hydrology 367(3–4): 165–176. Fodor, I. K. (2002). A survey of dimension reduction techniques, Technical report, Center for Applied Scientiﬁc Computing, Lawrence Livermore National Laboratory. Gibbs, M. S., Morgan, N., Maier, H. R., Dandy, G. C., Nixon, J. B. & Holmes, M. (2006). Investigation into the relationship between chlorine decay and water distribution parameters using data driven methods, Mathematical and Computer Modelling 44(5-6): 485–498. Guyon, I. & Elisseeff, A. (2003). An introduction to variable and feature selection, The Journal of Machine Learning Reearch 3: 1157–1182. Hastie, T., Tibshirani, R. & Friedman, J. (2001). The Elements of Statistical Learning. Data Mining, Inference and Prediction, Springer Series in Statistics, Springer, New York. Izrailev, S. & Agraﬁotis, D. K. (2002). Variable selection for QSAR by artiﬁcial ant colony systems, SAR and QSAR in Environmental Research 13(3–4): 417–423. Kingston, G. B. (2006). Bayesian Artiﬁcial Neural Networks in Water Resources Engineering, Ph.d., The University of Adelaide. Kohavi, R. & John, G. (1997). Wrappers for feature selection, Artiﬁcial Intelligence 97(1–2): 273–324. Kwak, N. & Choi, C.-H. (2002). Input feature selection for classiﬁcation problems, IEEE Transactions on Neural Networks 13(1): 143–159. Le Cun, Y., Denker, J. S. & Solla, S. A. (1990). Optimal brain damage, Advances in Neural Information Processing Systems, Morgan Kaufmann, pp. 598–605. Maier, H. R. & Dandy, G. C. (2000). Application of neural networks to forecasting of surface water quality variables: Issues, applications and challenges, Environmental Modelling & Software 15(1): 101–124. Mallows, C. L. (1973). Some comments on Cp, Technometrics 15: 661–675. Marcoulides, G. A. & Drezner, Z. (2003). Model speciﬁcation searches using ant colony optimization algorithms, Structural Equation Modeling: A Multidisciplinary Journal 10(1): 154–164. May, R. J., Maier, H. R. & Dandy, G. C. (2009a). Data splitting for artiﬁcial neural networks using SOM-based stratiﬁed sampling, Neural Networks 23: 283–294. May, R. J., Maier, H. R. & Dandy, G. C. (2009b). Development of artiﬁcial neural networks for water quality modelling and analysis, in G. Hanrahan (ed.), Modelling of Pollutants in Complex Environmental Systems, Vol. 1, ILM Publications, London, UK, pp. 27–62. Miller, A. J. (1984). Selection of subsets of regression variables, Journal of The Royal Statistical Society. Series A. 147(3): 389–425. Olsson, J., Uvo, C. B., Jinno, K., Kawamura, A., Nishiyama, K., Koreeda, N., Nakashima, T. & Morita, O. (2004). Neural networks for forecasting rainfall by atmospheric downscaling, Journal of Hydraulic Engineering, ASCE 9(1): 1–12. Rodriguez, M. J., Serodes, J.-B. & Cote, P. (1997). Advanced chlorination control in drinking water systems using artiﬁcial neural networks, Water Supply 15(2): 159–168.

44

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Sarle, W. (1997). Neural network FAQ. Periodic posting to the Usenet newsgroup comp.ai.neural-nets. Schwarz, G. (1978). Estimating the dimension of a model, Annals of Statistics 6(2): 461–464. Scott, D. W. (1992). Multivariate density estimation: theory, practice and visualisation, John Wiley and Sons, New York. Shannon, C. E. (1948). A mathematical theory of communication, Bell System Technical Journal 27: 379–423. Sharma, A. (2000). Seasonal to interannual rainfall probabilistic forecasts for improved water supply management: Part 1 - a strategy for system predictor identiﬁcation, Journal of Hydrology 239: 232–239. Shen, Q., Jiang, J.-H., Tao, J.-C., Shen, G.-L. & Yu, R.-Q. (2005). Modiﬁed ant colony optimization algorithm for variable selection in QSAR modeling : QSAR studies of cyclooxygenase inhibitors, Journal of Chemical Information and Modeling 45(4): 1024–1029. Silverman, B. W. (1986). Density estimation for statistics and data analysis, Chapman and Hall, London. Sooﬁ, E. S. & Retzer, J. J. (2003). Information importance of explanatory variables, IEE Conference in Honor of Arnold Zellner: Recent Developments in the Theory, Method and Application of Entropy Econometrics., Washington. Specht, D. F. (1991). A general regression neural network, IEEE Transactions on Neural Networks 2(6): 568–576. Tibrishani, R. (1996). Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society. Series B (Methodological) 58(1): 267–288. Tikka, J. (2008). Input variable selection methods for construction of interpretable regression models, Ph.d., Helsinki University of Technology. Torkkola, K. (2003). Feature extraction by non-parametric mutual information maximization, Journal of Machine Learning Research 3: 1415–1438. Trappenberg, T. P., Ouyang, J. & Back, A. D. (2006). Input variable selection: mutual information and linear mixing measures, IEEE Transactions on Knowledge and Data Engineering 18(1): 37–46. Wold, H. (1966). Estimation of principal components and related models by iterative least squares, in P. Krishnaiah (ed.), Multivariate Analysis, Academic Press, New York.

3
Artificial Neural Networks and Efficient Optimization Techniques for Applications in Engineering
1Federal

Rossana M. S. Cruz1, Helton M. Peixoto2 and Rafael M. Magalhães3

Institute of Education, Science and Technology of Paraiba, João Pessoa, PB, 2Federal University of Rio Grande do Norte, Natal, RN, 3Federal University of Paraiba, Rio Tinto, PB, Brazil

1. Introduction
This chapter proposal describes some artificial neural network (ANN) neuromodeling techniques used in association with powerful optimization tools, such as natural optimization algorithms and wavelet transforms, which can be used in a variety of applications in Engineering, for example, Electromagnetism (Cruz, 2009), Signal Processing (Peixoto et al., 2009b) and Pattern Recognition and Classification (Magalhães et al., 2008). The application of ANN models associated with RF/microwave devices (Cruz et al., 2009a, 2009b; Silva et al., 2010a) and/or pattern recognition (Lopes et al., 2009) becomes usual. In this chapter, we present neuromodeling techniques based on one or two hidden layer feedforward neural network configurations and modular neural networks trained with efficient algorithms, such as Resilient Backpropagation (RPROP) (Riedmiller & Braun, 1993), Levenberg-Marquardt (Hagan & Menhaj, 1999) and other hybrid learning algorithms (Magalhães et al., 2008), in order to find the best training algorithm for such investigation, in terms of convergence and computational cost. The mathematical formulation and implementation details of neural network models, wavelet transforms and natural optimization algorithms are also presented. Natural optimization algorithms, which are stochastic population-based global search methods inspired in nature, such as genetic algorithm (GA) and particle swarm optimization (PSO) are effective for optimization problems with a large number of design variables and inexpensive cost function evaluation (Kennedy & Eberhart, 1995; R. Haupt & S. Haupt, 2004). However, the main computational drawback for optimization of nonlinear devices relies on the repetitive evaluation of numerically expensive cost functions (Haupt & Werner, 2007; Rahmat-Samii, 2003). Finding a way to shorten the optimization cycle is highly desirable. In case of GA, for example, several schemes are available in order to improve its performance, such as: the use of fast full-wave methods, micro-genetic algorithm, which aims to reduce the population size, and parallel GA using parallel computation (R. Haupt & S. Haupt, 2004; Haupt & Werner, 2007). Therefore, this chapter

46

Artificial Neural Networks

also describes some hybrid EM-optimization methods, using continuous-GA and PSO algorithms, blended with multilayer perceptrons (MLP) artificial neural network models. These methods are applied to design spatial mesh filters, such as frequency selective surfaces (FSSs). Moreover, the MLP model is used for fast and accurate evaluation of cost function into continuous GA and PSO simulations, in order to overcome the computational requirements associated with full wave numerical simulations. Wavelets and artificial neural networks (ANN) have generated enormous interest in recent years , both in science and practical applications (Peixoto et al., 2009c). The big advantage of using wavelets is the fact of these functions make a local behavior, not only in the frequency domain but also in the field space and time. ANNs are capable of learning from a training set, which makes it useful in many applications, especially in pattern recognition. The purpose of using wavelet transforms is to find an easier way to compress and extract the most important features present in images, thereby creating a vector of descriptors that should be used to optimize the pattern recognition by a neural network. The array of descriptors contains elements whose values accurately describe the image content, and should take up less space than a simple pixel by pixel representation. The greatest difficulty found in this process is the generation of this vector, where the image content of interest should be very well described, in order to show really relevant features. Thus, this chapter proposal also shows a way to use a multiresolution technique, such as the wavelet transforms, to perform filtering, compression and extraction of image descriptors for a later classification by an ANN. This chapter is organized in six sections. Section 2 presents the most important fundamentals of artificial neural networks and the methodology used for the investigated applications. In section 3, artificial neural networks are optimized using wavelet transforms for applications in image processing (extraction and compression). Section 4 presents an EM-optimization using artificial neural networks and natural optimization algorithms for the optimal synthesis of stop-band filters, such as frequency selective surfaces. Section 5 shows a modular artificial neural network implementation used for pattern recognition and classification. Finally, section 6 presents important considerations about the artificial neural network models used in association with efficient optimization tools for applications in Engineering.

2. Artificial Neural Networks
2.1 Fundamentals Rosenblatt (1958) perceptron is the most used artificial neuron in neural network configurations and is based on the nonlinear model proposed by McCulloch and Pitts (1943). In this model, neurons are signal processing units composed by a set of input connections (weights), an adder (for summing the input signals, weighted by the respective synapses of a neuron, constituting a linear combiner) and an activation function, that can be linear or nonlinear, as shown in Fig. 1(a). The input signals are defined as xi, i = 0, 1,…, Ni, whose result corresponds to the level of internal activity of a neuron netj, as defined in (1), where x0 = +1 is the polarization potential (or bias) of the neurons. The output signal yj is the activation function response ( ) to the activation potential netj, as shown in (2) (Silva et al., 2010b).

Artificial Neural Networks and Efficient Optimization Techniques for Applications in Engineering

47 (1) (2)

net j

Ni i 0

w ji xi

yj

(net j )

For a feedforward neural network (FNN), the artificial neurons are set into layers. Each neuron of a layer is connected to those of the previous layer, as illustrated in Fig. 1(b). Signal propagation occurs from input to output layers, passing through the hidden layers of the FNN. Hidden neurons represent the input characteristics, while output neurons generate the neural network responses (Haykin, 1999). Modular artificial neural network is based on a principle commonly used: divided to conquer. This concept aims to divide a large and complex task in a set of sub-tasks that are easier to be solved. The modular artificial neural network could be defined, in summary, as a set of learning machines, also called experts, whose decisions are combined to achieve a better answer than the answers achieved individually, that is, a machine with a better performance. In the past few years, one of the main areas of learning machine is the characterization of methods capable to design this kind of machines. There are two types of these machines: static and dynamic structures. Modular neural networks, as seen in Fig. 1(c), is a dynamic type. The input signal is used by the gating network to design the global response. An advantage of modular artificial neural networks, when compared with other neural networks, is the learning speed. The machine learning process is accelerated in case of problems where it is observed a natural decomposition of data at simple functions. To develop the modular machine architecture and to implement the experts, it is usual to apply multilayer perceptrons (MLP) neural networks. 2.2 Methodology Generally, the design of a neural network is composed by three main steps: configuration how layers are organized and connected; learning – how information is stored; generalization how neural network produces reasonable outputs for inputs not found in the training (Haykin, 1999). In this work, we use feedforward and modular neural networks associated with supervised learning to develop neural network models. In the computational simulation of supervised learning process, a training algorithm is used for the adaptation of neural network synaptic weights. The instantaneous error e(n), as defined in (3), represents the difference between the desired response, d(n), and the neural network output, z(n), at the iteration n, corresponding to the presentation of the nth training pattern [x(n);(d(n)] (Silva et al., 2010b). e(n) z(n) d(n)

(3)

Supervised learning can be illustrated through the block diagram of Fig. 2(a) and has as objective the minimization of the mean square error E(t), given in (4), where the index t represents the number of training epochs (one complete presentation of all training examples, n = 1, 2,…, N, where N is the total number of examples, called an epoch) (Silva et al., 2010b).

48

Artificial Neural Networks

E(t )

1 N 1 e(n)2 N n 12

(4)

Currently, there are several algorithms for the training of neural networks that use different optimization techniques (Peixoto et al., 2009a). The most popular training algorithms are those derived from backpropagation algorithm (Rumelhart et al., 1986). Among the family of backpropagation algorithms, the RPROP shows to be very efficient in the solution of complex electromagnetic problems. In this work, the stop criteria are defined in terms of the maximum number of epochs and/or the minimum error and the activation function used was the sigmoid tangent (Haykin, 1999). After training, the neural network is submitted to a test, in order to verify its capability of generalizing to new values that do not belong to the training dataset, for example, parts of the region of interest where there is not enough knowledge about the modeled device/circuit. Therefore, the neural network operates like a “black box” model that is illustrated in Fig. 2(b) (Silva et al., 2010b). Resilient backpropagation algorithm is a first-order local adaptive learning scheme. The basic principle of RPROP is to eliminate the harmful influence of the partial derivative size in the weight update. Only the sign of the derivative is considered to indicate the direction of the weight update, whp , as given in (5). hp (t ),

if if

E (t ) 0 whp E (t ) 0 whp

whp (t )

hp (t ),

(5)

0, elsewhere

The second step of RPROP algorithm is to determine the new update-values hp(t). This is based on a sign-dependent adaptation process, similar to the learning-rate adaptation shown by Jacobs (1988). The changes in the weight size are exclusively determined by a weight ‘update-value’, hp, as given in (6).

hp (t hp (t ) hp (t hp (t

1), 1),

E (t 1) whp E (t 1) whp

E (t ) 0 whp E (t ) 0 whp

(6)

1), elsewhere

Here, the following RPROP parameters were employed: 50. values were restricted to the range 10-6 hp

1.2 and

0.5. The update-

Artificial Neural Networks and Efficient Optimization Techniques for Applications in Engineering

49

Fig. 1. (a) Nonlinear model of an artificial neuron; (b) FNN configuration with two hidden layers; (c) Extended modular neural network configuration with K experts

50

Artificial Neural Networks

Fig. 2. (a) Block diagram of supervised learning; (b) neural network “black box” model

3. Artificial Neural Networks optimization using wavelet transforms
This section introduces the main concepts about wavelet transforms and some of their most important features used to optimize artificial neural networks: the extraction of characteristic information descriptors in order to improve training and pattern classification mechanisms. Moreover, a model that can be used to solve various problems related to pattern recognition is presented and a neural classifier is implemented to validate the importance of such optimization. Transforms are widely used mathematical tools to understand and analyze different signal behaviors. The objective of this analysis is to extract important information (or features) that can essentially represent the signal from some decomposition or transformation performed on it.
3.1 Wavelet transforms Wavelet transforms have generated enormous interest from scientists, resulting in the development of applications in various areas, such as computer vision (Wang et al., 2009), seismology (Parolai et al., 2008), radar (Masnadi-Shirazi et al., 2009), astronomy (Ottensamer et al., 2008), image compression (Bhatia et al., 2009), signal filtering (Vimal et al., 2009), system optimization (Pinto, 2009) and many others. In general, the major advantage of using wavelet transforms is the possibility of applying it to non-stationary signals, which allows the study of function local behaviors, in both frequency and time-scale domains. 3.1.1 Advantages of use Traditional methods of signal analysis based on Fourier transform can determine all the frequencies present in the signal, however its relationship to the time domain does not exist. To overcome this problem, it was created the Gabor transform (or STFT - Short Time Fourier Transform); the main idea of this transform is to introduce a new measure of local frequency as the local transformation observed the signal through a narrow window within which the signal remains nearly stationary (Oliveira, 2007). The problems in time and frequency domain resolution are result of a physical phenomenon known as Heisenberg's uncertainty principle (it is impossible to know the exact frequency and time that a signal occurs). This

Artificial Neural Networks and Efficient Optimization Techniques for Applications in Engineering

51

phenomenon is independent of the transformation used (Oliveira, 2007). Therefore, the wavelet transform was developed as an alternative to the Gabor transform to solve the resolution problem. Wavelets are mathematical transformations that separate signals in different components and extract each one of them with a resolution apropriated to its corresponding scale. According to the transformation characteristics, there is the Continuous Fourier Transform (CFT), that can be expressed as:
F( w ) f ( t )e j 2 ft

dt

(7)

Knowing the spectrum F(w) of a signal, it is possible to obtain it in time domain, using the inverse transform concept, according to (8): f (t ) 1 F ( w )e 2 j 2 ft

dw

(8)

On the other hand, the Continuous Wavelet Transform (CWT) is given by:
CWT ,a f (t ) 1 a t a dt

(9)

and its corresponding inverse can be expressed according to (10): f (t ) 1 C CWT ( a , b ) 1 a t a dadb a2 (10)

where (t ) is the mother wavelet, respectively.

and a are the translation and scale parameters,

3.1.2 Discrete wavelets The continuous wavelet transform is calculated by performing continuous translations and scalings of a function over a signal. In practice, this transformation is not feasible because it is necessary to make endless translations and scalings, requiring much time, effort and computational redundancy. Discrete wavelets were introduced to overcome this problem and therefore they are used in this work. They are not translated or scaled continuously but in discrete steps, which is achieved from a modification of the continuous wavelet: s, (t )

1 s

t s j t k 0 s0 j s0

(11)

j , k (t )

1 j s0

(12) is the translation factor,

where j and k are integers; s0 > 1 is a fixed dilation parameter; which depends on the dilation factor.

0

52

Artificial Neural Networks

Generally, it is chosen s0 = 2 in order to have a sampling frequency called dyadic sampling 0 = 1 is chosen for the temporal sampling, also dyadic This can be shown in (13) (Oliveira, 2007): j , k (t )

2j

2jt k

(13)

When discrete wavelets are used to analyze a signal, the result is a series of wavelet coefficients, also called the series of wavelet decomposition (Oliveira, 2007). As a wavelet can be viewed as a bandpass filter, the scaled wavelet series can be seen as a set of bandpass filters, with a Q factor (set of filters fidelity factor). In practice, there is a discretized wavelet, with upper and lower limits for translations and scales. The wavelet discretization, associated with the idea of passing the signal through a filter set, results in the well known subband coding (Oliveira, 2007).
3.1.3 Multiresolution analysis Mathematical transformations are used in a dataset to obtain additional information not available in the primitive data model. For example, it may be necessary to use a transformation that detects changes in color tones of a pixel neighborhood and its corresponding spatial location, and even that efficiently transposes these changes in a multiresolution space (Castleman, 1996). The multiresolution analysis using wavelet transforms has become increasingly popular with the release of JPEG-2000 standard (Weeks, 2007) and consists of a signal processing strategy where it is used a set of specialized filters to extract signal information, such as the range of frequencies present in it and their location as a function of the signal duration at different resolutions (Castleman, 1996). A brief description of the multiresolution analysis enables to display two functions responsible for generating the entire wavelet system: the scale function and the primary wavelet (or mother wavelet). The term mother comes from the fact that functions with different sizes are used in the process of transformation and all of them are originated from a specific or mother wavelet. Scale function j , k and the primary wavelets j , k are considered orthogonal to follow the condition shown in (14): j ,k

x

j ,k

x dx

0

(14)

Z corresponds to the scale function parameter and k Z corresponds to the k translation of j in relation to the scale function and the primary wavelet, given by j = 0 2 and k = 0, respectively. Both scale and wavelet functions are defined in the set of real (R) numbers, by scalings and translations of the mentioned functions. The translation parameter corresponds to time information in the transform domain and the scaling parameter is the process of compression and expansion of the signal (Mallat, 2009). In Fig. 3 is shown an example of a scale function and a primary Haar wavelet. Therefore, one can say that multiresolution analysis using discrete time wavelets corresponds to successive band-pass filtering, through which signals are decomposed at

where j

Artificial Neural Networks and Efficient Optimization Techniques for Applications in Engineering

53

Fig. 3. Scale function and the primary Haar wavelet each step, in terms of approximation and details. Fig. 4 illustrates this procedure being applied to an input image, where filters are used in successive rows and columns, creating the new scales. The reverse process, which performs the sum of subspaces, can reconstruct the original image. The dilation equations expressed by h(k) represent the low-pass filters that generate the approximations of the original image. However, the translation equations g(k) represent the high-pass filters and are responsible for obtaining the details of the original image. The decomposition of detail functions consists of details on vertical (highpass filter at the rows and low-pass filters at the columns), details on horizontal (low-pass filter at the rows and high-pass filters at the columns), details on diagonal (high-pass filter at the rows and columns).

Fig. 4. Wavelet decomposition Then, using an input image with scale of (j+1), with m rows and n columns, it is shown in Fig. 5 a two level image decomposition example. The original image and its approximations are the lighter areas of the picture. The other three remaining subpictures correspond to the three detail functions of the original image. In the second step, the lightest part of the picture is decomposed again, generating a new image approximation and three new detail subpictures. Thus, a twice smaller scale image

54

Artificial Neural Networks

Fig. 5. Multiresolution representation of an image was generated. Fig. 6 shows an initial image and the degree of refinement obtained with the wavelet transform. This form of decompose and reconstruct images can be implemented quickly and effectively, because of the use of wavelet transforms.

Fig. 6. Multiresolution analysis of an image
3.2 Wavelet pre-processing Pre-processing step aims to improve the performance of the next steps. Then, the refinement of these data that will be used for training and classification are of fundamental importance. Designing a neural classifier consists of choosing architecture, training algorithm, quality of training dataset, among other aspects that must be optimized, in order to reduce the time used in network training at the same time that the accuracy of the results is increased. This optimization can be performed using wavelet transforms. The block diagram described in Fig. 7 shows that the data input used by the neural network may be submitted to a wavelet pre-processing or not. Here, both the situations are analyzed in parallel.

Fig. 7. Block diagram of a neural network system

Artificial Neural Networks and Efficient Optimization Techniques for Applications in Engineering

55

3.3 Neural network implementation A simple but effective neural classifier has been implemented at this step, using the classic aspects of this type of project. It is important to know that even robust classifiers as SVM (Support Vector Machine) can not function properly without a pre-processing. Fig. 8 shows a sample training set used in the experiment. This consists of 50 images containing the vowels A, E, I, O, U, with slight 2D dislocation and no noise.

Fig. 8. Training dataset model Fig. 9 shows a sample of the test dataset (composed by 25 images), where there is the inclusion of uncorrelated noise to images up to four levels, aiming to difficult the neural network generalization.

Fig. 9. Test dataset model Here, the neural network configuration used the following parameters: a multilayernperceptron neural network; the Levenberg-Marquardt training algorithm; number of epochs (150) or Error (10e-5) as stop criteria; two hidden layers and the sigmoid tangent as activation function. The results are shown in Table 01, according to two categories of pre-processing: without wavelet - larger training set (designed pixel by pixel) and higher time to neural network learning; with wavelet (Daubechies, level 2) - smaller and more efficient training dataset due to the choice of better wavelet descriptors (higher intensity), which provides a faster network learning. It is important to observe that the stop criterion was the number of epochs while the error was kept around 10e-4 and the generalization result was about 92%, which enabled to recognize images of the vowels a, e, i, o, u with small 2D dislocation levels (= number of patterns. The one-shot learning algorithm is as follows: 1. Choose an arbitrary value greater than 4 for the self-feedback coefficient of each selffeedback unit ( wD 4 ). j 2. Regarding the selected value for w D calculate following terms for each self-feedback j unit: bj1 2 ln wD j wD j 2 wD j 2
(j 1,..., m)

wD j

4

wD j

wD j 2 wD j

4

bj2

2 ln wD j

wD j

4

wD j

wD j 2 wD j

4

(9)

3.

4.

To use the maximum capacity of the SFNN, adjust the value of parameter bj as bj122.04 ; s51>28.00 ; s61>33.97 ; ; s12= 2005) } EVALUATE ?k FOR ?researcher { ?h = ris:hIndex([?doc,?docCite]); } ORDER BY DESC(?k) Fig. 7. GRASQL query that ranks researchers by k index using MEDLINE abstracts.

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

181

%keyword rix:hasWord ?year rip:publishYear ?doc rdf:type rip:hasDocument

?researcher rdf:type %documentSet

%researcher

Fig. 8. RDF graph pattern specified in the WHERE clause in Fig. 7. RDF graphs. Further, the k index, namely the number of documents ?doc without duplication for each researcher ?r, is computed by calling the statistical function ris:countDistinct in the EVALUATE clause.

@prefix rio: @prefix rip: @prefix rix: @prefix ris: @prefix rdf: @let %keyword “type 2 diabetes” @let %documentSet rio:MEDLINE @let %geneSet rio:MouseGene SELECT ?gene ?p WHERE { ?gene rip:hasDocument ?docGene ; rip:hasDocument ?docIntersection ; rdf:type %geneSet . ?docKey EXT:rix:hasWord %keyword ; rdf:type %documentSet . ?docIntersection EXT:rix:hasWord %keyword ; rdf:type %documentSet . ?docGene rdf:type %documentSet . ?docAll rdf:type %documentSet . } EVALUATE ?p FOR ?gene { ?p = ris:statisticTest#FisherExactTest(?a,?b,?c,?d) ; ?a = count(DISTINCT ?docIntersection) ; ?b = count(DISTINCT ?docKey)-?a ; ?c = count(DISTINCT ?docGene)-?a ; ?d = count(DISTINCT ?docAll)-?a-?b-?c } ORDER BY ?p Fig. 9. GRASQL query for a direct search using Fisher’s exact test as a method of computing the statistical significance of the intersection ?docIntersection.

182

Artificial Neural Networks - Methodological Advances and Biomedical Applications

rdf:type rdf:type rdf:type

?docKey ?docIntersection

rix:hasWord

%keyword

rix:hasWord rip:hasDocument

%documentSet

rdf:type

?docGene ?docAll

rip:hasDocument

?gene

rdf:type

%geneSet

Fig. 10. RDF graph pattern satisfying the condition described in the WHERE clauses shown in Figs. 9 and 12. 3.2 Statistical tests in the PosMed search Our method discovers entities significantly related to a user’s keyword by using the documents associated with the entities. In this study, we use ‘entity’ (or ‘document’) to clearly denote that the RDF name is a biomedical entity (or a document). The simplest method for discovering entities is (1) full-text search over documents to find those containing the user’s keyword, and then (2) obtaining entities associated with the documents found. This process is notated here as user' s keyword document entity . To compute the significance of the association between each entity and a keyword, we have introduced a statistical test based on the number of shared documents. More concretely, for each entity, the search engine first generates a 2×2 contingency table consisting of the number of documents a. matching both the keyword and the entity, b. matching the keyword but not matching the entity, c. not matching the keyword but matching the entity and d. matching neither the keyword nor the entity. Then, the engine applies a statistical test to the contingency table and computes a P-value, or the significance of the test. Finally, all resultant entities are ranked by their P-values. We call the discovery method described above a direct search, which is described by the query in Fig. 9. The statistical function ris:statisticalTest#FisherExactTest computes the a b P-value by constructing a 2×2 contingency table with its four arguments a ,b ,c and d, c d and applies Fisher’s exact test to the table. The simple method of evaluating the query shown in Fig. 9 is a sequential evaluation of the WHERE, EVALUATE and ORDER BY clauses in this order. In this method, the WHERE clause is evaluated to obtain all RDF graphs satisfying the condition in the WHERE clause. Figure 10 shows the RDF graph pattern with all variables and constants appearing in the WHERE clause. In practice, since the number of RDF graphs matching the pattern in Fig. 10 may be huge, this simple method of evaluation requires the implementation of an optimisation mechanism to achieve a functional language processor. Figure 11 is a chart that includes a Venn diagram of MEDLINE abstracts; it shows the relationship between each subset of MEDLINE abstracts and other RDF entities. This figure shows the primitive data structure for query searching as a set of relationships between each subset of MEDLINE abstracts and other entities such as %keyword and ?gene, rather than the relationships between each MEDLINE abstract and the other entities. In our example, to compute ?p, only four subsets—?docAll, ?docKey,

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

183

?docIntersection and ?docGene —of MEDLINE abstracts are necessary; they can be obtained by a specialised document search method such as the full-text search technique. Since this approach does not require a huge RDF graph space for computing the statistical significance, it opens up a new possibility for realising a practical GRASQL language processing system. Furthermore, the results of statistical analysis can be stored as a named graph using the CONSTRUCT statement instead of the SELECT statement. Figure 12 shows a CONSTRUCT query that generates RDF graphs with blank nodes as shown in Fig. 11. The generated named graph can be efficiently used as input data in SPARQL as well as in GRASQL. Statistical tests can also be used in a search to indirectly generate the associations between entities and a keyword via entity–entity relationships associated with documents. A typical example of entity–entity relationships is the co-citation frequencies of the entities in documents. The significance of the association between two entities can be computed by a statistical test of the number of documents, similar to a direct search. That is, for each entity– entity relationship, a P-value is computed using a 2×2 contingency table that contains the number of documents a. matching both entities, b. matching the first entity but not matching the second, c. not matching the first entity but matching the second and d. matching neither entity. The entity–entity relationship can be obtained as a set of RDF triples using the query shown in Fig. 13.
%documentSet
?docAll rdf:type rix:hasWord

%keyword ripSingleSearch:hasWord ?docKey ?docIntersection ?docGene

Evaluate the statistical significance of the intersection ripSngleSearch:hasPValue Construct new triples

?p ripSingleSearch:hasEntity rip:hasDocument

?gene

rdf:type

%geneSet

Fig. 11. Statistical diagram showing the relationships among the entities specified by the query in Fig. 12. New RDF triples are constructed by the query’s CONSTRUCT statement. We realise an inference search for the connection user ' s keyword document entity document entity by applying entity–entity relationships to the entities resulting from a single association search. The P-value Pd of the associated entity is computed by

Pd

1

1 Ps 1 Pr

(1)

where Ps is the P-value of the first direct search, and Pr is the P-value of the second association search of the entity–entity relationship. Furthermore, several search connections, user' s keyword document entity1 document entity 2 , which reach the same entity entity2 via a different entity entity1, may be obtained. In this case, the P-value of the resultant entity entity2 can be computed by

184

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Pentity2

i

Pi , entity2

(2)

where Pi , entity2 , 1 i n are the P-values of n connections that finally reach entity 2 . This model is based on the idea that a solution containing several connections may be more important than others. Another method is selecting the best connection by choosing the smallest P-value. In this case, the equation,

Pentity2 is applied for computing the P-value.

min( Pi , entity 2 ) i (3)

@prefix @prefix @prefix @prefix @prefix @prefix

rio: rip: rix: ris: rdf: ripSigleSearch: @let %keyword “type 2 diabetes” @let %documentSet rio:MEDLINE @let %geneSet rio:MouseGene CONSTRUCT { [] ripSingleSearch:hasEntity ?gene ; ripSingleSearch:hasWord %keyword ; ripSingleSearch:hasPValue ?p . } WHERE { ?gene rip:hasDocument ?docGene ; rip:hasDocument ?docIntersection ; rdf:type %geneSet . ?docKey EXT:rix:hasWord %keyword ; rdf:type %documentSet . ?docIntersection EXT:rix:hasWord %keyword ; rdf:type %documentSet . ?docGene rdf:type %documentSet . ?docAll rdf:type %documentSet . } EVALUATE ?p FOR ?gene { ?p = ris:statisticTest#FisherExactTest(?a,?b,?c,?d) ; ?a = count(DISTINCT ?docIntersection) ; ?b = count(DISTINCT ?docKey)-?a ; ?c = count(DISTINCT ?docGene)-?a ; ?d = count(DISTINCT ?docAll)-?a-?b-?c }

Fig. 12. GRASQL query including a CONSTRUCT statement, which is used to save the results of the statistical analysis described in the WHERE and EVALUATE clauses into a set of RDF graphs. In the CONSTRUCT statement, as in SPARQL, a blank node [ ] is used to describe the relationships among ?gene, %keyword and?p.

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

185

3.3 GRASQL representation of gene prioritisation in PosMed To describe the semantics of gene prioritisation in PosMed more precisely, we will write the direct search and inference search patterns shown in Figs. 3(A) and 3(B), respectively, in GRASQL. A GRASQL query for a direct search is written in Fig. 12. For convenience in enumerating examples, we first assume that the named graph http://omicspace.riken.jp/GRASQL/single/Mm/MEDLINE from the direct search obtained by the query in Fig. 11 is generated. We also assume that the named graph http://omicspace.riken.jp/GRASQL/relation/Mm/MEDLINE of entity–entity relationships obtained by the query in Fig. 13 is generated. Using these two named graphs, we write a query for an inference search of the connection user' s keyword document entity document entity , as shown in Fig. 14.

@prefix @prefix @prefix @prefix @prefix

rio: rip: ris: rdf: ripInference: @let %documentSet rio:MEDLINE @let %geneSet rio:MouseGene CONSTRUCT { [] ripInference:hasEntity1 ?gene1 ; ripInference:hasEntity2 ?gene2 ; ripInference:hasPValue ?p . } WHERE { ?gene1 rip:hasDocument ?docGene1 ; rip:hasDocument ?docIntersection ; rdf:type %geneSet . ?gene2 rip:hasDocument ?docGene2 ; rip:hasDocument ?docIntersection ; rdf:type %geneSet . ?docGene1 rdf:type %documentSet . ?docGene2 rdf:type %documentSet . ?docIntersection rdf:type %documentSet . ?docAll rdf:type %documentSet . } EVALUATE ?p FOR ?gene1 ?gene2 { ?p = ris:statisticTest#FisherExactTest(?a,?b,?c,?d) ; ?a = count(DISTINCT ?docIntersection) ; ?b = count(DISTINCT ?docKey)-?a ; ?c = count(DISTINCT ?docGene)-?a ; ?d = count(DISTINCT ?docAll)-?a-?b-?c }

Fig. 13. GRASQL query that builds RDF triples of co-citation relationships of mouse genes from MEDLINE abstracts.

186

Artificial Neural Networks - Methodological Advances and Biomedical Applications

@prefix rio: @prefix ris: @prefix rdf: @prefix ripSingleSearch: @prefix ripInference: @let %keyword “type 2 diabetes” @let %geneSet rio:MouseGene SELECT ?gene2 ?gene1 ?p ?pTotal FROM NAMED FROM NAMED WHERE { ?x ripInference:hasEntity2 ?gene2 ; ripInference:hasEntity1 ?gene1 ; ripInference:hasPValue ?pInference . ?y ripSingleSearch:hasEntity ?gene1 ; ripSingleSearch:hasWord %keyword ; ripSingleSearch:hasPValue ?pSingle . } EVALUATE ?p FOR ?gene1 ?gene2 { ?p = 1-(1-?pSingle)(1-?pInference) } EVALUATE ?pTotal FOR ?gene2 { ?pTotal = ris:multiPValue(?p) } ORDER BY ?pTotal ?p

Fig. 14. GRASQL query for inference search for connection %keyword ?gene1 ?gene2 using named graphs generated by CONSTRUCT statements in advance. In this query, the two EVALUATE clauses are evaluated sequentially in the order of their appearance. In the example, P-value ?p for each pair (?entity1, ?entity2) is computed, and then P-value ?pTotal Total for each entity ?entity2 is computed. Finally, by evaluating the ORDER BY clause, the solutions of 4-tuples (?entity1, ?entity2, ?p, ?pTotal) are sorted by ?pTotal and ?p. The function ris:multiPValue in the second EVALUATE clause is an implementation of Equation 2. Furthermore, ris:minPValue is an implementation of Equation 3 that does not appear in this article.

4. Data preparation and implementation
4.1 Data sources Currently, PosMed employs more than 20 million documents including MEDLINE (title, abstract and MeSH term), genome annotation, phenome information, PPI, co-expression, localisation, disease, drug and metabolite records (Table 1). 4.2 High-accuracy manual curation for generating semantic links from genes to documents To develop a set of document databases for our original search engine for PosMed, we developed a method of mapping between genes and documents based on an NER (Leser & Hakenberg, 2005) technique that extracts named entities such as genes from a document.

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

187

A.

PosMed
No. of Data sources Data Contents documents MEDLINE title, abstract and MeSH 18 295 132 MEDLINE term 12 911 BRMM Mouse phenotypes 21 136 OMIM 35 731 HsPPI Genetic disorder descriptions Protein-protein interaction Reference Coletti & Bleich, 2001 Masuya et al., 2007 Amberger et al., 2009 Makino & Gojobori, 2007 Matthews et al., 2009 Blake et al., 2009 Dwinell et al., 2009 Wain et al., 2002 Shinbo et al., 2006

MEDLINE Mouse mutant OMIM HsPPI REACTOME Mouse gene record Rat gene record Human gene record Metabolite record Drug record Disease record RIKEN researcher record Total

10 761 REACTOME Biological pathways 58 768 MGI 36 634 RGD 35 362 HGNC Gene descriptions (annotations) Gene descriptions (annotations) Gene descriptions (annotations)

18 045 KNApSAcK Metabolite descriptions 1 015 Original data Drug descriptions 1 911 Original data Disease descriptions Names of researchers appear as 8 603 Original data authors in MEDLINE 18 534 098

B.

PosMed-plus
No. of Data sources Data Contents documents MEDLINE title, abstract and 18 295 132 MEDLINE MeSH term Microarray based co-expression 44 082 ATTED-II prediction Experimentally validated 8 404 SUBA-2 subcellular localisation 24 418 AtPID Protein-protein interaction RIKEN Arabidopsis Phenome 214 RAPID Information DB Phenotype informations from 1 697 TAIR TAIR 1 784 Literature Manually collected original data 1 712 RFLP marker RAP-DB 15 623 SSR marker Original Homologue genes between 1 553 922 data Arabidopsis and rice 33 003 TAIR, UniProt Gene descriptions (annotations) Gene descriptions (annotations) Reference Coletti & Bleich, 2001 Obayashi et al., 2009 Heazlewood et al., 2007 Cui et al., 2008 Kuromori et al., 2006 Swarbreck et al., 2008 Harushima et al., 1998 McCouch et al., 2002 Hanada et al., 2008 Swarbreck et al., 2008 ; UniProt Consortium, 2009 Rice Annotation Project, 2008

MEDLINE At coexpression At localisation At PPI

At phenotype Rice markers Homologus genes Arabidopsis gene record Rice gene record Total

29 389 RAP-DB 20 009 380

Table 1. Data descriptions for (A) PosMed and (B) PosMed-plus.

188

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Since false-positive relationships may arise from a primitive NER method that simply checks for the appearance of a name in a document, we instead employ a full-text search engine for NER, with logical queries defined as a list of names or words related to a gene concatenated with logical operators such as AND, OR and NOT. Specifically, as a base query we computationally collected all the synonym names for each gene from The Arabidopsis Information Resource (TAIR) and UniProt, connected these synonyms with the logical OR operation and added ‘Arabidopsis’ with the AND operation. Using these base queries, we performed a full-text search against a set of documents including MEDLINE title, abstract and MeSH terms (Coletti & Bleich, 2001). To reduce false-positive hits and true-negative hits, we carefully edited these queries manually through trial and error by performing a full-text search for each trial against the document set. For example, to detect all MEDLINE documents for the AT1G03880 (cruciferin B, CRB) gene while eliminating false-positive hits with the homonym ‘CRB’, which represents ‘chloroplast RNA binding’, we defined the following query: (‘AT1G03880’ OR ‘CRU2’ OR ‘CRB’ OR ‘CRUCIFERIN 2’ OR ‘CRUCIFERIN B’) AND (‘Arabidopsis’) NOT (‘chloroplast RNA binding’). This curation method is effective for updating with the latest publications. Once we curate a query, the query can be reused to extract gene–document relationships by performing a fulltext search against those new document sets.
4.3 Implementation PosMed was developed as a web-oriented tool based on a client-server model in which users access the system with conventional web browsers. However, we recommend using Microsoft Internet Explorer 8 or later or Firefox 3 or later for Windows, and Safari 4 or later or Firefox 3 or later for Macintosh. The core software component GRASE must execute a search process by very rapidly interpreting a GRASQL query program. To develop GRASE, we employed Apache Lucene, a rapid full-text search engine with a rich query language, for testing the predicate rix:hasWord. Since a search process can be executed for each target entity in parallel, we use nine distributed computers to realise a high-throughput search. Therefore, we distributed the data for each entity; i.e. MEDLINE abstracts and mouse gene–mouse gene relationship data associated with each distributed mouse gene and researcher are distributed among the computers to achieve a parallel search.

5. Applications of PosMed
We describe examples illustrating the power of PosMed and PosMed-plus below.
5.1 General usage of PosMed 5.1.1 Search with user-specified keywords and chromosomal intervals A typical application of PosMed is searching with user-specified keywords and chromosomal intervals suggested by linkage analysis. As an example, we retrieved diabetes- or insulin-related genes in the chromosomal interval from 90 Mbp to 140 Mbp on chromosome 1 in the mouse genome (Fig. 15(A)). In this example, PosMed retrieved candidate genes ranked by the statistical significance between the user’s keyword and each gene. Although PosMed found > 470 000 documents, it returned results in 0,865 s. Users can download all the candidate genes together with the associated gene annotations by using the ‘download rank list’ button in the blue box on the left (Fig. 15(D)). PosMed also supports an expert mode that allows users to select possible search paths and confirm the number of resulting genes for each search path. Clicking on a gene name listed in the gene search result

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

189

(A)

Select genomic interval graphically

(D)

(B)

(C)

Clicking here displays Fig. 16

Fig. 15. Example search result for mouse genes against the query keyword ‘diabetes or insulin’ and the genomic interval between 90 Mbp and 140 Mbp on chromosome 1 in the NCBIm 37 genome. Users can construct queries at the top of the output display (A). To select a genomic interval visually, PosMed cooperates with the Flash-based genomic browser OmicBrowse. The ‘All Hits’ tab (B) shows a list of selectable document sets to be included in the search. As a default parameter, PosMed sets ‘Associate the keyword with entities co-cited within the same sentences’. If the total number of candidate genes is less than 20, PosMed will automatically change this to ‘Associate the keyword with entities cocited within the same document’ to show more candidates (B). Search results are ranked in (C). Users can download at most 300 candidate genes and their annotations from (D).

190

Artificial Neural Networks - Methodological Advances and Biomedical Applications

(A)

(B) (E) (C)

(D)

Fig. 16. Detailed document screen in PosMed. This page shows document sets supporting both the Adipor1 gene ranked fifth in Fig. 15(C) and the Adipoq gene. Gene descriptions are shown in (A). Users can select the type of documents from the mouse mutant, HsPPI, MEDLINE mouse gene record or REACTOME in (B). The bar chart represents the number of related documents per year. Red and blue indicate the number of documents with and without a user-specified keyword, respectively. All documents are shown at (D). The Adipor1-related genes are listed in (E).

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

191

page shown in Fig. 15(B) reveals the supporting evidence for each candidate gene. To confirm the expression pattern of candidate genes with a genome browser, we provide a link to our genome browser OmicBrowse (Matsushima et al., 2009; Toyoda et al., 2007) from the gene location (Fig. 15(C)). OmicBrowse covers genome versions for mouse, human, rat, Arabidopsis and rice, and each genome is mapped to omic-type databases and a total of 344 data sources.
5.1.2 Search with phenotypic keywords PosMed also allows users to discover genes related to phenotypic keywords. For example, if users search on the keyword ‘rumpled leaves’ in Arabidopsis, PosMed-plus shows four known cases via the direct search and one new candidate gene via the inference search. For the four known cases, PosMed-plus shows the link to the RIKEN Arabidopsis Phenome Information Database (RAPID), and users can confirm the phenotypes by looking at pictures. PosMed-plus also shows the evidence documents in the inference path to the AT1G51500 candidate gene. In this case, AT1G51500 is retrieved via the AT1G17840 gene, which is one of the four known genes found in the direct search. They are highly connected with co-expression, PPI and co-citation data. 5.1.3 Reference search with gene IDs It is difficult to retrieve all the appropriate references based on gene names because of the wide variation in synonyms. Moreover, sometimes the same abbreviated names are used for functionally different genes, causing false-positive hits. In PosMed, we carefully extracted these gene–reference relationships manually, as described above. Therefore, users can retrieve the curated results with the gene ID (e.g. MGI code and AGI code) even if the abstracts do not contain the gene ID itself. 5.1.4 Search for omics data As shown in Fig. 16, PosMed integrates various data such as gene annotations, mouse mutant records and human PPIs. Users can select any document set (the default setting is to search everything) and retrieve the required data, all within the same interface. PosMed links not only to the original databases but also to OmicBrowse, which also assists users in accessing and downloading various omics data. 5.2 In silico positional cloning after QTL analysis in rice To evaluate the efficiency of PosMed with a concrete example, we confirmed whether PosMed (PosMed-plus in this rice example) could successfully retrieve correct genes that have been identified by qualitative trait locus (QTL) analysis. Three examples are described below. Ren et al. (2005) isolated the SKC1 gene and through QTL analysis found that it encoded an Na+-selective transporter. In this example, we need to prioritise candidate genes without the functionally related keyword ‘transporter’. Instead of the functional keyword, we retrieved genes with the phenotypic keyword ‘salt tolerance’ and selected the genomic interval between the markers C955 and E50811 on chromosome 1. PosMed-plus returned the Os01g0307500 (cation transporter family protein) gene with a high ranking. This is because the keyword ‘salt tolerance’ was mapped to the sodium ion transmembrane transporter gene AT4G10310, and Os01g0307500 was suggested as a homologue of AT4G10310. Using a no-pollen type of male-sterile mutant (xs1), Zuo et al. (2008) revealed that mutant microspores are abnormally condensed and agglomerated to form a deeply stained cluster at the late microspore stage. This halts the microspore vacuolation process, and therefore,

192

Artificial Neural Networks - Methodological Advances and Biomedical Applications

the mutant forms lack functional pollen. This mutation is controlled by a single recessive gene, VR1 (vacuolation retardation 1), which is located between the molecular markers RM17411 and RM5030 on chromosome 4. We searched for candidate genes with the phenotypic keyword ‘sterility’ in the suggested chromosome region. PosMed-plus suggested the Os04g0605500 gene (similar to calcium-transporting ATPase) as the homologue of the Arabidopsis calcium-transporting ATPase, AT3G21180. Since Schiøtt et al. (2004) found that mutation of AT3G21180 results in partial male sterility, we conclude that PosMed-plus found an appropriate candidate. Lastly, Zhang et al. (2008) found a male sterility mutant of anther dehiscence in advance, add(t), between the markers R02004 and RM300 on chromosome 2. In this search, PosMed-plus returned RNA-binding region RNP-1, Os02g0319100 and disease-resistance protein family protein Os02g0301800, with strong homology with Arabidopsis genes. PosMed-plus retrieved the Os02g0319100 gene as a homologue of Arabidopsis mei2-like (AML) protein 5, AT1G29400. As supporting evidence, Kaur et al. (2006) showed that multiple mutants of all the AML genes displayed a sterility phenotype. The other candidate gene, Os02g0301800, was derived via an inference search. First, PosMed-plus retrieved the keyword ‘sterility’ in a document describing the AT2G26330 gene. Next, AT2G26330 was linked to AT5G43470 as supported by three co-citations. Finally, Os02g0301800 was returned as a homologue of AT5G43470. PosMed-plus originally suggested the Os02g0301800 gene because AT2G26330 is linked to the keyword ‘sterility’ in a document. However, this document states that AT2G26330 causes aberrant ovule development and female-specific sterility. Since Zhang et al. (2008) focused on male sterility, we conclude that Os02g0319100 is the appropriate candidate.
5.3 Other example results In RIKEN’s large-scale mouse ENU mutagenesis project, PosMed was used to prioritise genes and has contributed to the successful identification of more than 65 responsible genes (Masuya et al., 2007). PosMed is also used by researchers worldwide and has successfully narrowed the candidate genes responsible for a specific function after QTL analysis (Kato et al., 2008; Moritani et al., 2006). 5.4 Further usage We here introduced PosMed as a web tool for assisting in the prioritisation of candidate genes for positional cloning. Using the search engine GRASE, we also implemented inference-type full-text search functions for metabolites, drugs, mutants, diseases, researchers, document sets and databases. For cross-searching, users can select ‘any’ for the search items at the top right on the PosMed web page. Since this system can search various omics data, we named it OmicScan. In addition to English, GRASE accepts queries in Japanese and French. More advanced usage of PosMed is explained in the PosMed tutorial available at http://omicspace.riken.jp/tutorial/HowToUseGPS_Eng.pdf.

6. Discussion and conclusion
To use not only well-formed knowledge in RDF but also non-well-formed document data on the Semantic Web, we have introduced statistical concepts into the existing RDF query language SPARQL using a literature mining technique for searching a vast number of documents written in a natural language. The core data structure in our method is that documents are linked with each entity accurately associated by NER with human refinement, namely manual curation. The advantages of this simple structure are as follows.

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

193

Facility of keyword selection: An arbitrary keyword appears in at least one document. Thus, a user can choose a keyword that is not necessarily related to an entity the user wants to find. Open-ended extensibility of documents: A new document can be added to the system if it is associated with at least one existing entity. Documents about an entity written from various viewpoints enrich knowledge so that the entity can be linked to the user’s keyword. Open-ended extensibility of entities: A new entity can be added if at least one document associated with it exists. Therefore, entities of different categories can be introduced, which allows association search among them. Open-ended extensibility of semantic knowledge: Existing biomedical data in RDF format can be introduced directly into a GRASQL query. Thanks to these advantages, PosMed can support various types of heterogeneous omics knowledge. PosMed has been widely used to prioritise candidate genes after QTL analysis in species including mouse and Arabidopsis and to successfully identify responsible genes. Our approach is novel compared to gene prioritisation systems such as BIOTLA (Hristovski et al., 2005), Manjal (Sehgal & Srinibasan, 2005) and LitLinker (Yetisgen-Yildiz & Pratt, 2006), since PosMed is based on P-values computed by Fisher’s exact test via tables of numbers of documents and used as correlation scores between a user’s keyword and the resulting genes for ranking. Our future work will include data extension of PosMed with not only well-formed omics knowledge in RDF but also non-well-formed document data on the Semantic Web using the statistical concepts of GRASQL.

7. References
Adie, E.; Adams, R.; Evans, K.; Porteous, D. & Pickard, B. (2006). SUSPECTS: enabling fast and effective prioritization of positional candidates. Bioinformatics, Vol. 22, 773-774 Aerts, S.; Lambrechts, D.; Maity, S.; Van Loo, P.; Coessens, B.; De Smet, F.; Tranchevent, L.; De Moor, B.; Marynen, P.; Hassan, B.; Carmeliet, P. & Moreau, Y. (2006). Gene prioritization through genomic data fusion. Nat. Biotechnol., Vol. 24, 537-544 Amberger, J.; Bocchini, C.; Scott, A. & Hamosh, A. (2009). McKusick’s Online Mendelian Inheritance in Man (OMIM). Nucleic Acids Res., Vol. 37, D793-D796 Ashburner, M.; Ball, C.A.; Blake, J.A.; Botstein, D.; Butler, H.; Cherry, J.M.; Davis, A.P.; Dolinski, K.; Dwight, S.S.; Eppig, J.T.; Harris, M.A.; Hill, D.P.; Issel-Tarver, L.; Kasarskis, A.; Lewis, S.; Matese, J.C.; Richardson, J.E.; Ringwald, M.; Rubin, G.M. & Sherlock, G. (2000). Gene ontology: tool for the unification of biology. Nat. Gene., Vol. 25, 25-29 Berners-Lee, T.; Hendler, J. & Lassila, O. (2001). The Semantic Web. Sci. Am., Vol. 284, 34-43 Blake, J.; Bult, C.; Eppig, J.; Kadin, J. & Richardson, J. (2009). The Mouse Genome Database genotypes::phenotypes. Nucleic Acids Res., Vol. 37, D712-D719 Coletti, M. & Bleich, H. (2001). Medical subject headings used to search the biomedical literature. J. Am. Med. Inform. Assoc., Vol. 8, 317-323 Cui, J.; Li, P.; Li, G.; Xu, F.; Zhao, C.; Li, Y.; Yang, Z.; Wang, G.; Yu, Q. & Shi, T. (2008). AtPID: Arabidopsis thaliana protein interactome database - an integrative platform for plant systems biology. Nucleic Acids Res., Vol. 36, D999-D1008

194

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Dwinell, M.; Worthey, E.; Shimoyama, M.; Bakir-Gungor, B.; DePons, J.; Laulederkind, S.; Lowry, T.; Nigram, R.; Petri, V.; Smith, J.; Stoddard, A.; Twigger, S.; Jacob, H. & the RGD Team. (2009). The Rat Genome Database 2009: variation, ontologies and pathways. Nucleic Acids Res., Vol. 37, D744-D749 The Gene Ontology Consortium. (2006). The Gene Ontology (GO) project in 2006. Nucleic Acids Res., Vol. 34, D322-D326 GeneSniffer. Available From http://www.genesniffer.org Hanada, K.; Zou, C.; Lehti-Shiu, M.; Shinozaki, K. & Shiu, S. (2008). Importance of lineagespecific expansion of plant tandem duplicates in the adaptive response to environmental stimuli. Plant Physiol., Vol. 148, 993-1003 Harushima, Y.; Yano, M.; Shomura, A.; Sato, M.; Shimano, T.; Kuboki, Y.; Yamamoto, T.; Lin, S.Y.; Antonio, B.A.; Parco, A.; Kajiya, H.; Huang, N.; Yamamoto, K.; Nagamura, Y.; Kurata, N.; Khush, G.S. & Sasaki, T. (1998). A high-density rice genetic linkage map with 2275 markers using a single F2 population. Genetics, Vol. 148, 479-494 Heazlewood, J.; Verboom, R.; Tonti-Filippini, J.; Small, I. & Millar, A. (2007). SUBA: the Arabidopsis subcellular database. Nucleic Acids Res., Vol. 35, D213-D218 Hirsch, J.E. (2005). An Index to Quantify an Individual’s Scientific Research Output. Proc. Natl. Acad . Sci . USA, Vol. 102, 16569-16572 Hristovski, D.; Peterlin, B.; Mitchell, J.A. & Humphrey, S.M. (2005). Using literature-based discovery to identify disease candidate genes. Int. J. Med. Inform., Vol. 74, 289-298 Kato, N.; Watanabe, Y.; Ohno, Y.; Inoue, T.; Kanno, Y.; Suzuki, H. & Okada, H. (2008). Mapping quantitative trait loci for proteinuria-induced renal collagen deposition. Kidney Int., Vol. 73, 1017-1023 Kaur, J.; Sebastian, J. & Siddiqi, I. (2006). The Arabidopsis-mei2-like genes play a role in meiosis and vegetative growth in Arabidopsis. Plant Cell, Vol. 18, 545-559 Kobayashi, N. & Toyoda, T. (2008). Statistical search on the Semantic Web. Bioinformatics, Vol. 24, 1002-1010 Köhler, S.; Bauer, S.; Horn, D. & Robinson, P. (2008). Walking the interactome for prioritization of candidate disease genes. Am. J. Hum. Genet., Vol. 82, 949-958 Kuromori, T.; Wada, T.; Kamiya, A.; Yuguchi, M.; Yokouchi, T.; Imura, Y.; Takabe, H.; Sakurai, T.; Akiyama, K.; Hirayama, T.; Okada, K. & Shinozaki, K. (2006). A trial of phenome analysis using 4,000 Ds-insertional mutants in gene-coding regions of Arabidopsis. Plant J., Vol. 47, 640-651 Leser, U. & Hakenberg, J. (2005). What makes a gene name? Named entity recognition in the biomedical leterature. Brief Bioinform., Vol. 6, 357-369 Makino, T. & Gojobori, T. (2007). Evolution of protein-protein interaction network. Genome Dyn., Vol. 3, 13-29 Makita, Y.; Kobayashi, N.; Mochizuki, Y.; Yoshida, Y.; Asano, S.; Heida, N.; Deshpande, M.; Bhatia, R.; Matsushima, A.; Ishii, M.; Kawaguchi, S.; Iida, K.; Hanada, K.; Kuromori, T.; Seki, M.; Shinozaki, K. & Toyoda, T. (2009). PosMed-plus: an intelligent search engine that inferentially integrates cross-species information resources for molecular breeding of plants. Plant Cell Physiology, Vol. 50, 1249-1259 Manola, F. & Miller, E. (2004). RDF Primer. World Wide Web Consortium, Recommendation REC-rdf-primer-20040210. Available from http://www.w3.org/TR/2004/REC-rdf-primer-20040210/

Prioritising Genes with an Artificial Neural Network Comprising Medical Documents to Accelerate Positional Cloning in Biological Research

195

Masuya, H.; Yoshikawa, S.; Heida, N.; Toyoda, T.; Wakana, S. & Shiroishi, T. (2007). Phenosite: a web database integrating the mouse phenotyping platform and the experimental procedures in mice. J. Bioinform. Comput. Biol., Vol. 5, 1173-1191 Masuya, H.; Makita, Y.; Kobayashi, N.; Nishikata, K.; Yoshida, Y.; Mochizuki, Y.; Doi K.; Takatsuki, T.; Waki, K.; Tanaka, N.; Ishii, M.; Matsushima, A.; Takahashi, S.; Mizoguchi, R.; Kozaki K.; Furuichi, T.; Kawaji, H.; Wakana, S.; Nakamura, Y.; Yoshiki, A.; Murata, T.; Fukami-Kobayashi, K.; Mohan, S.; Ohara, O.; Hayashizaki, Y.; Obata, Y. & Toyoda, T. (2010). The RIKEN integrated database of mammals. Nucleic Acids Res., (in press) Matsushima, A.; Kobayashi, N.; Mochizuki, Y.; Ishii, M.; Kawaguchi, S.; Endo, T.A.; Umetsu, R., Makita, Y. & Toyoda, T. (2009). OmicBrowse: a Flash-based high-performance graphics interface for genomic resources. Nucleic Acids Res., Vol. 37, Web Server Issue, W57-W62 Matthews, L.; Gopinath, G.; Gillespie, M.; Caudy, M.; Croft, D.; de Bono, B.; Garapati, P.; Hemish, J.; Hermjakob, H.; Jassal, B.; Kanapin, A.; Lewis, S.; Mahajan, S.; May, B.; Schmidt, E.; Vastrik, I.; Wu, G.; Birney, E.; Stein, L. & D’Eustachio, P. (2009). Reactome knowledgebase of human biological pathways and processes. Nucleic Acids Res., Vol. 37, D619-D622 McCouch, S.; Teytelman, L.; Xu, Y.; Lobos, K.; Clare, K.; Waltam, M.; Fu, B.; Maghirang, R.; Li, Z.; Xing, Y.; Zhang, Q.; Kono, I.; Yano, M.; Fjellstrom, R.; DeClerck, G.; Schneider, D.; Cartinhour, S.; Ware, D. & Stein, L. (2002). Development and mapping of 2240 new SSR markers for rice (Oryza sativa L.). DNA Res., Vol. 9, 199-207 Moritani, M.; Togawa, K.; Yaguchi, H.; Fujita, Y.; Yamaguchi, Y.; Inoue, H.; Kamatani, N. & Itakura, M. (2006). Identification of diabetes susceptibility loci in db mice by combined quantitative trait loci analysis and haplotype mapping. Genomics, Vol. 88, 719-730 Noruzi, A. (2005). Google Scholar: The New Generation of Citation Indexes. Libli, Vol. 55, 170-180 Obayashi, T.; Hayashi, S.; Saeki, M.; Ohta, H. & Kinoshita, K. (2009). ATTED-II provides coexpressed gene networks for Arabidopsis. Nucleic Acids Res., Vol. 37, D987-D991 Prud’Hommeaux, E. & Seaborne, A. (2008). SPARQL Query Language for RDF. World Wide Web Consortium, Recommendation REC-rdf-sparqlquery-20080115. Available from http://www.w3.org/TR/2008/REC-rdf-sparql-query-20080115/ Ren, Z.; Gao, J.; Li, L.; Cai, X.; Huang, W.; Chao, D.; Zhu, M.; Wang, Z.; Luan, S. & Lin, H. (2005). A rice quantitative trait locus for salt tolerance encodes a sodium transporter. Nat. Genet., Vol. 37, 1141-1146 Rice Annotation Project (2008). The Rice Annotation Project Database (RAPDB): 2008 update. Nucleic Acids Res., Vol. 36, D1028-D1033 Schiött, M.; Romanowsky, S.; Baekgaard, L.; Jakobsen, M.; Palmgren, M. & Harper, J. (2004). A plant plasma membrane Ca2+ pump is required for normal pollen tube growth and fertilization. Proc. Natl. Acad . Sci . USA, Vol. 101, 9502-9507 Seelow, D.; Schwarz, J. & Schuelke, M. (2008). GeneDistiller - distilling candidate genes from linkage intervals. PLoS ONE, Vol. 3, e3874, 537-544 Sehgal, A.K. & Srinivasan, P. (2005). Manjal - A Text Mining System for MEDLINE. Proceedings of the 28th Annual International ACM SIGIR., Salvador, Brazil, 680

196

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Shinbo, Y.; Nakamura, Y.; Altaf-Ul-Amin, Md.; Asahi, H.; Kurokawa, K.; Arita, M.; Saito, K.; Ohta, D.; Shibata, D. & Kanaya, S. (2006). KNApSAcK: A Comprehensive SpeciesMetabolite Relationship Database, In: Plant Metabolomics, Biotechnology in Agriculture and Forestry, Vol. 57, Saito, K.; Dixon, R.A. & Willmitzer, L., 165-184, Springer Verlag, Berlin Swarbreck, D.; Wilks, C.; Lamesch, P.; Berardini, T.Z.; Garcia-Hernandez, M.; Foerster, H.; Li, D.; Meyer, T.; Muller, R.; Ploetz, L.; Radenbaugh, A.; Singh, S.; Swing, V.; Tissier, C.; Zhang, P. & Huala, E. (2008). The Arabidopsis Information Resource (TAIR): gene structure and function annotation. Nucleic Acids Res., Vol. 36, D1009-D1014 Thornblad, T.; Elliott, K.; Jowett, J. & Visscher, P. (2007). Prioritization of positional candidate genes using multiple web-based software tools. Twin Res. Hum. Genet., Vol. 10, 861-870 Toyoda, T.; Mochizuki, Y.; Player, K.; Heida, N.; Kobayashi, N. & Sakaki, Y. (2007). OmicBrowse: a browser of multidimensional omics annotations. Bioinformatics, Vol. 23, 524-526 UniProt Consortium (2009). The Universal Protein Resource (UniProt) 2009. Nucleic Acids Res., Vol. 37, D169-D174 Van Driel, M.; Cuelenaere, K.; Kemmeren, P.; Leunissen, J.; Brunner, H. and Vriend, G. (2005). GeneSeeker: extraction and integration of human disease-related information from web-based genetic databases. Nucleic Acids Res., Vol. 33, W758-W761 Wain, H.; Lush, M.; Ducluzeau, F. & Povey, S. (2002). Genew: the human gene nomenclature database. Nucleic Acids Res., Vol. 30, 169-171 Yetisgen-Yildiz, M. & Pratt, W. (2006). Using statistical and knowledge-based approaches for literature-based discovery. J. Biomed. Inform., Vol. 39, 600-611 Yoshida, Y.; Makita, Y.; Heida, N.; Asano, S.; Matsushima, A.; Ishii, M.; Mochizuki, Y.; Masuya, H.; Wakana, S.; Kobayashi, N. & Toyoda, T. (2009). PosMed (Positional Medline): prioritizing genes with an artificial neural network comprising medical documents to accelerate positional cloning. Nucleic Acids Res., Vol. 37, Web Server issue, W147-W152 Zhang, Y.; Li, Y.; Zhang, J.; Shen, F.; Huang, Y. & Wu, Z. (2008). Characterization and mapping of a new male sterility mutant of anther advanced dehiscence (t) in rice. J. Genet. Genomics, Vol. 35, 177-182 Zuo, L.; Li, S.; Chu, M.; Wang, S.; Deng, Q.; Ding, L.; Zhang, J.; Wen, Y.; Zheng A. & Li, P. (2008). Phenotypic characterization, genetic analysis, and molecular mapping of a new mutant gene for male sterility in rice. Genome, Vol. 51, 303-308

10
Artificial Neural Networks Technology to Model and Predict Plant Biology Process
1Applied

Plant and Soil Biology, Dpt. Plant biology and Soil Science, Faculty of Biology, University of Vigo, 36310 Vigo 2Dpt. Pharmacy and Pharmaceutical Technology, Faculty of Pharmacy, University of Santiago. 15782 Santiago de Compostela, Spain

Pedro P. Gallego1, Jorge Gago1 and Mariana Landín2

1. Introduction
The recent and significant technological advances applied to biology places the researchers in front of an unprecedented new influx of large data set from different levels as genomics, transcriptomics, proteomics, metabolomics and ionomics (Hirai et al., 2004; Belostotsky & Rose, 2005; Schauer & Fernie, 2006; Kliebenstein, 2010). Thousands of data sets including millions of measurements have been generated, and moreover, most are freely available for plant researchers worldwide from plant specific databases, as for example the whole sequencing of different plant genomes like rice, Arabidopsis, poplar, papaya, grapevine and others... (Jaillon et al., 2007; Ming et al., 2008; Brady & Provart, 2009). There is a wide concern of integrating molecular, cellular, histological, biochemical, genetic and physiological information in plant biology (Katagiri, 2003; Thum et al., 2003; Trewavas 2006; Boone et al., 2007; Álvarez-Buylla et al., 2007) and also in other related fields such as crop improvement (Hammer et al., 2002), ecology (Hilbert & Ostendorf, 2001; Jimenez et al., 2008) and biological engineering (Huang, 2009). Biological processes are both time variant and nonlinear in nature, and their complexity can be understood as the composition of many different and interacting elements governed by non-deterministic rules and influenced by external factors (Coruzzi et al., 2009, Gago et al., 2009). Commonly, most of biological interactions cannot be elucidated by a simple stepwise algorithm or a precise formula, particularly when the data set are complex, noisy, vague, uncompleted or formed by different kind of data (Prasad & Dutta Gupta, 2008; Gago et al., 2010a). It is important to point out that many times the behaviour of a biological system over a time period is difficult to understand and interpret and additionally, genetic and environmental factors show a very high degree of intra- and inter-individual variability, yielding a wide spectrum of biological responses (Karim et al., 1997; Guégan et al., 1998). The Scientific community agrees with the idea that plant biology requires more efforts in developing platforms to integrate multidimensional data and to derive models for describing biological interactions in plants (Kitano, 2002; Hammer et al., 2004; Struik et al., 2005; Tardieu, 2003; Yuan et al., 2008; Brady & Provart, 2009). In this sense, more efforts are recommended to shift our view from a reductionist way to a systems-level view. This concept can be illustrated by the Coruzzi & co-workers (2009) example of the painting “La Grande Jatte” by the

198

Artificial Neural Networks - Methodological Advances and Biomedical Applications

pointillist artist George Seurat. If one stands near the image each of its strokes can be distinguish but cannot appreciate the beauty of the whole composition. Only from far away is possible to admire the beautiful sunset over the Seine in Paris. Most researchers are really on top of their topic, but far away from the whole view of the global subject. However, the future in plant research requires a broader view of biological plant systems, from the new available molecular and cellular discoveries to the whole-plant improvements (Kirschner, 2005; Yuan et al., 2008) or the performance at the crop level (Kitano, 2002; Wang et al., 2002), and even to the agroecological level (Jiménez et al., 2008; Huang, 2009). Performing a meta-analysis of the data set of a whole system is not an easy task. Hammer & collaborators (Hammer et al., 2004) expressed the present research requirement, comparing it to the paths of navigation the early sailors needed to determine to be able to arrive at their destiny with accuracy. In the 18th century this scientific problem was solved with the development of appropriate tools to predict the longitude and latitude in the middle of the oceans. Thus, these authors proposed an interesting conclusion: researchers need the equivalent to sailor’s tools to navigate across the different levels of biological organization from gene to phenotype. As Tardieu (Tardieu, 2003) has pointed out that as in all “marriages of convenience” the first contact is always difficult between the ones having the tools, the modellers, with an essential mathematical and physical background, and the others, in this case the plant biologists, having the data and the knowledge in plant physiology (from molecular to whole plant and from individual cells to whole populations) who necessarily have to work together to obtain this exciting new challenge. Since the spectacular development of computers, researchers have been attempting to create non-biological entities that can imitate human level of performance. Such attempts have manifested in the emergence of a cognitive approach termed as artificial intelligence (AI) (Legg & Hutter, 2007). In 1956, John McCarthy defined this term as "the science and engineering of making intelligent machines", and currently textbooks define this field of computer science as "the study and design of intelligent agents". AI achieved higher popularity in the 90s and early 21st century was introduced as a new tool in different scientific and technical fields (Russell & Norvig, 2003). Since then, successful studies have been carried out using different techniques, such as artificial neural networks, fuzzy logic and genetic algorithms, which can combine and complement in multiple ways and have been used in many industrial and commercial applications (Taylor, 1996) such as: character speech or image recognition (Hussain & Kabuka, 1998; Ma & Klorasani, 2004), chemical research (Cartwright, 1993; Zupan & Gasteiger, 1993), process modelling and control (Lennox et al., 2001), and in pharmaceuticals (Rowe & Roberts, 1998; Shao et al., 2006) or biomedicine (Hudson & Cohen, 2000). Finally, also since the late 90s artificial networks have been used in some biological areas, such as ecology or environmental sciences (Lek & Guégan, 1999; Hilbert & Ostendorf, 2001; Huang, 2009). Since the basis for understanding the theoretical and practical approaches, of these technologies, to the development of models and their applications to specific problems, has been review elsewhere (Müller et al., 1995; Rowe & Roberts, 1998; Hudson & Cohen, 2000; Huang, 2009) and in other chapters of the present book. Therefore, the purpose of this chapter is to review the topics relevant to AI technology, mainly genetic algorithms and fuzzy logic, and more extensively, neural networks for the integration of multidimensional data into models (networks) and the application of these models in addressing questions (decision making) in plant biology. As plant biology researchers are not commonly used to these new technologies; we will begin with a brief introduction for the readers on their fundamentals in order to facilitate the understanding of its applicability.

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

199

2. Analysis of biological data
Data from plant biology are inherently complex. The types of data must be taken into account since they influence the kind of analysis to be carried out. Normally, plant biology data can be classified as binary data, those with only two possible responses, generally yes/no, i.e. survival (alive–dead); discrete data, which have more than two responses (which can take one of a finite set of values), i.e. number of proliferated shoots or flowers in an inflorescence: 0, 1, 2, 3…); continuous data, which have any response (which can take any of an infinite number of values, i.e. weight rates of the proliferated shoots or the flowers…). Less frequent data in the plant biological process are image data ( -glucuronidase (GUS) and green fluorescence protein (GFP) histological analysis in transformation experiments); temporal data (a particular sequence of events: phenological development of fruits; duration of time for the fruit growth; or fruit fresh weight gain or loss per month), time series data (chilling hours in different seasons and their effect on bud formation) or fuzzy data (some processes or physiological states in plant science are described by linguistic tags. Using fuzzy data: for example, the embryo developmental stages such as the different sets can be explained as globular, torpedo, heart; or the callus colour, can be classified as: brown, brownish, yellowish during a plant in vitro culture callogenesis process; see fuzzy logic section for a complete description). All these types of data can be included in a neural networks systems database, but the precision and accuracy of the number of data must be taken into account. Over years, experimental designs and statistics have been important research tools for the plant researchers. Conventional analytical tools including logistic regression (for binomial and multinomial); Poisson regression (for discrete data), analysis of variance (ANOVA) for data only normally or approximately normally distributed continuous data to extract conclusions from data and to understand biological process have generally been used (Mize et al., 1999; Gago et al., 2010a). These techniques have allowed many questions to be solved, however many shortcomings can also been pointed out. Relevant difficulties are found when researchers need to consider a large data set with different kinds of data at the same time, to model the whole process studied or when the non-idealities of plant science processes do not conform (Hammer et al., 2004; Prassad & Dutta Gupta, 2008; Gago et al., 2010a). Such problems need a different sort of intelligence and connectionism. The kind of approach that shows through nodes and connection diagrams of the interaction and integration of multiple components in organisms is possible at present using computational models (Rumelhart & McClelland, 1986; Yuan et al., 2008; Huang, 2009).

3. Modelling plant biology process
Since the purpose of modelling is to increase our understanding of a plant science process (for example: providing understanding of the regulatory networks controlling developmental, physiological or other processes in plants) plant biology models to approach the complexity of these processes are needed. To solve this requirement, an abstraction is needed which is able to organize the factors (inputs) and the parameters measured (outputs) into a functional model. These principles are really useful to approach the biological complexity, by abstracting and focusing on the relevant factors to obtain a broad view of the whole system and become essential for the understanding principal questions and decision support. To explain the concept of how the plant biologists can approach the biological complexity, an interesting example of what is needed of the abstraction is as follows: in 1736 an old mathematical problem “The seven

200

Artificial Neural Networks - Methodological Advances and Biomedical Applications

bridges of Königsberg”, a city in Prussia (actually Kaliningrado in Russia), was negatively resolved by Leonhard Euler, and meant the beginning of the mathematical graph theory. This city was crossed by the Pregel river and had two islands in the middle, connected by seven bridges to each other and to the mainland. Would it be possible to find a way through the city by crossing each bridge once? Euler reorganised the problem in abstract terms: giving no importance to the city or if the bridge was made of wood or stone... only to reducing all the information to nodes/vertex (land masses) and edges/link (bridges). The answer: “It would be not possible”; however, Euler´s success was in reducing the problem to the significant inputs/factors and their relationships avoiding irrelevant data for the final purpose and to promote a suitable analysis. Mathematical and computational models have been dramatically increasing in recent years in biological related sciences as in ecology (Stollenwerk et al., 2001; Anderson & Jensen, 2005), environmental and biodiversity conservation (Williams et al., 2004), epidemiology and pathogenesis (Brauer & Castillo-Chavez, 2001), genetic and biotechnology (Bar-Joseph et al., 2003), evolution (Nijhout et al., 2003) and animal (Schuster et al., 2005; Tracqui, 2006) or plant biology (Kovalenko & Riznichenko, 2007). There is a wide variety of mathematical and computational models. Normally they are based in algorithms developed by theoretical and applied mathematicians, physics and/or bioinformatics, or for people with a firm background in theoretical biology, biological chemistry, mathematical biology in collaboration with informatics, computer science, physics, and/or engineering departments. In conclusion, mathematical modelling of biological systems generally requires a wide variety of methods and skills from multiple disciplines (de Vries et al., 2006). For developing any mathematical model it is necessary to follow different steps: a) identification of the problem or process to be simulated, controlled and/or optimized; b) selection of data, variables (input and outputs), and very importantly, what is the model for; c) introduction of the accurate and precise data according from each variable, selection of equations (mainly algorithms) and the type of model: white box (all information is mostly available) or black box (no a priori information is available). If the black box model is chosen, some parameters can be used to fit the model to the system (in neural network the optimization of those parameters is called training); and d) model evaluation (normally cross-validation), to check the distances between the observed and predicted data (which should be as low as possible). The validity of the model is not only about if it fits well or not to empirical observations, but also about its ability to provide new insight which is partially occluded in the data, and which can not be known from direct observation or from statistical data analysis of the process. If the purpose is not achieved, the model is probably unnecessary, time consuming and useless. Another important issue to take into account is the quality of data. The model will be excellent if data are excellent. Models do not produce miracles. Bad quality data (non accurate, disperse or non precise) are not the best option to fit a model, especially if they are selected for a black box model, as neural networks are.

4. Artificial Neural Networks
Artificial Neural Networks (ANNs) are computational systems that simulate biological neural networks and they have been widely described in previous chapters of this book and in detail elsewhere (Russell & Norvig, 2003; Rowe & Roberts, 2005), but to better understand their application to plant biology it is pertinent to briefly review the way it is believed to function.

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

201

The main elements of biological neural networks are the neurons, electrochemical excitable cells that can receive signals or stimuli from other neurons via synapse connection (Fig. 1). The stimuli is received through the dendrites and transmitted to the cell body. If the stimulus is intense enough, the neuron generates another stimulus that is transmitted along the axon to the next neuron via synapses.

Fig. 1. Basic comparison between a biological neuron and an artificial neuron. X= input variable; W=weight of in input; = internal threshold value; f=transfer function. The artificial neural network architecture is an interconnected assembly of individual processing elements called perceptron: “single nodes” or “artificial neurons” (Fig. 1). Each artificial neuron receives one or more inputs from neighbouring nodes, process the information and produces an output to be transmitted to the next node. The strengths of connections between two units are called “weights” which must be defined by the computational approach to solve or interpret a given problem (Takayama et al., 1999). While computing the output, the input information (Xi) is weighed either positively or negatively. The computational approach must also assign an internal threshold value ( ) to simulate the output action. At each node, the input values (Xn) are multiplied by their associate weight (Wn) to give a result, which is adjusted by its threshold value. The output is then determined using the non-linear weighted sum as the argument in a function “f” termed transfer function or activation function (Fig. 1; eq. 1). yi=f ( Wn-Xni)

(1)

Among the functions that can be applied: linear, hyperbolic tangent or radial basis form, etc is the sigmoid function (eq. 2) and is the most commonly used. Sigmoid function, f(yi), is conducted to the following layer as an output value. Alpha is a parameter relating to the shape of the sigmoid function. Non-linearity of the sigmoid function is strengthened with an increase in . f(yi)=1/[1+exp(- yi)] (2)

202

Artificial Neural Networks - Methodological Advances and Biomedical Applications

By connecting several artificial neurons (many hundreds of thousands) or simple nodes a complex artificial neural network can be obtained. Figure 2 illustrates the topology of one of the most popular and successful network architectures, a multilayer perceptron (MLP) networks, consisting of three simple layers; one input layer, one output layer and with just one hidden layer.

Fig. 2. A multilayer perceptron with one hidden layer. ANNs is able to “learn” an approximate non-linear relationship between inputs and outputs using algorithms designed to alter the strength (weights) of the connections in the network to produce a desired signal flow. This “training” process is defined as a search process for the optimized set of weight values which can minimize the squared error between the data predicted by the model and the experimental data in the output layer (Takayama et al., 1999). The ability of the network to memorize and process the information lies in the weights assigned to the inter-node connections, which determines the conductivity through the network. When the computed output is unacceptable, compared with experimental output, a back propagation process starts to modify several setting parameters (also called the learning rule) until the network attains a good generalization of the problem domain (Prasad & Dutta Gupta, 2008). The difference between actual and predicted outputs is usually quantified by means of an error function similar to those used in statistics. In different research fields, it has been proposed that the performance of a well-designed MLP network is comparable to that achieved by classical statistical techniques (Rowe & Roberts, 1998) and therefore, suitable for a wide range of applications including: classification (Glezakos et al., 2010), pattern recognition (Frossyniotis et al., 2008), prediction on time series (Müller et al., 1995) interpolation (Gago et al., 2010a), and modelling complex systems with non-linear behaviour (Karim et al., 1997; Mehrota et al., 2008; Gago et al., 2010b, c) It is important to point out that the strength of ANNs lays on its ability in detecting and quantifying complex non-linear relationships between inputs and outputs as well as its capability on generalizing distorted or partially occluded patterns (Taylor, 1996; Shao et al.,

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

203

2006). This powerful technology has also limitations, mainly related to the difficulties of interpreting the results in simple form or to elaborate formal reasoning or extract general rules (Colbourn, 2003). Complex “black box” models derived from ANNs technologies from a large data set with an important number of inputs could be difficult to analyze using general 2D plots or even 3D graphs. In order to avoid those limitations ANNs are usually combined with other AI techniques as genetic algorithms or fuzzy logic technology giving hybrid systems the results of which are easier to interpret and generalize (Plumb et al., 2005; Shao et al., 2006; Shao et al., 2007; Gago et al., 2010d). 4.1 Artificial Neural Networks as an alternative to traditional statistics As pointed out by Mize and coworkers (1999), an extensive review of the literature shows that the use of correct statistical tools is not widespread in plant science. Many papers can be found where authors treat discrete or binomial variables as continuous variables by using the ANOVA. Moreover, when analyzed correctly, usually multiple comparison tests are performed to determine which of the factors studied have a significant effect on certain parameter or process. The use of modeling and optimization techniques is even more restricted, generally being reduced to fit data to a specific functional form (linear or quadratic). This is because, in many cases, the plant researcher lacks the appropriate mathematical background for the analysis and his/her interaction with experts in statistics is not very fruitful. In this situation the use of a different technology as ANNs can be of great help. Recent studies have demonstrated that AI technologies show the same or even better performance than traditional statistics for modelling complex non linear relationships hidden in the data and offer superior prediction powers (Landin et al., 2009 and references therein; Gago et al., 2010a). From a formal point of view ANNs show several advantages over statistics: a) they can process different types of data together (continuous, binomial, discrete); b) they can be used to produce complex models without the previous knowledge of the functional form of dependence, so they can discover subtle relationships in the data; c) they do not require specific experimental design, being capable of using incomplete data, data acquired during a series of trial-and-error experiments or even historical data (Colbourn, 2003; Colbourn and Rowe, 2005). Moreover, the use of ANNs does not require a specialized background and is a friendly technology, easy to use, that allows the modelling process with a limited number of experiments and costs, and makes inference of the combination of factors studied possible to obtain the best result (Gago et al., 2010a). 4.2 Applications of neural networks to plant biology There are not many references in the literature on the applications of ANN to plant biology. This fact is more relevant, if we compare with other related areas as pharmaceutical science of important research in the last few years (Achanta et al., 1995; Colbourn, 2003; Takayama et al., 1999; Shao et al., 2006; Landín et al., 2009), ecology (Guégan et al., 1998; Hilbert et al., 2001; Adriaenssens et al., 2004) or agriculture (Huang, 2009 and references therein). Pioneer studies in plant science deal with the use of AI technology to improve and/or optimize biotechnology processes production. An early work (Fukuda et al., 1991) explored the use of artificial neural networks to recognize live or dead plant cells by image processing. Other authors have also investigated the capabilities of image analysis of

204

Artificial Neural Networks - Methodological Advances and Biomedical Applications

somatic embryos developmental stage by neural networks (Uozumi et al., 1993) and image analysis of shoot length of regenerated rice callus using a hybrid (artificial neural networks/fuzzy logic) technology (Honda et al., 1997). Additionally, works dealing with modelling the bioproduction of Ginjo sake were carried out using a fuzzy neural network in order to control the processes in bioreactors (Hanai et al., 1997). During the last twenty years there has been an increasing interest on this technology in the agricultural and biological engineering fields (Huang, 2009). Firstly, it was applied to model food quality (Whittaker et al., 1991; Eerikäinen et al., 1993) or fruit colour (Thai & Shewfelt, 1991) and, more recently, to manage herbicide application (Yang, 2003), yield estimation (Kaul et al., 2005; Khazaei et al., 2008), and water stress (Ondimu & Murase, 2008). Neural networks were used for modelling crop yields on the basis of environmental conditions and pest control treatments in order to improve production. The optimization of pesticide concentration and periods of treatments to be used has a great impact on the costs and toxic residual levels of agriculture products (Jiménez et al., 2008). Health and economy are the important issues in the agricultural production nowadays. Different authors also established the relationship between the factors and crop yield for corn, sugar beet, soybean and winter wheat in order to help on decision-making processes (Kehagias et al., 1998; Kaul et al., 2005; Green et al., 2007; Jiménez et al., 2008). Other authors have described the usefulness of ANNs for modelling the distribution of vegetation in past, present and future climates coupled with GIS (geographic information system). They have provided worthy contributions to understanding and conservation of these areas, especially when more detailed biogeographical data were available (Hilbert & Ostendorf, 2001). More recently, detection of plant viruses has been carried out through a Bioelectric Recognition Assay (BERA) method in combination with neural networks (Frossyniotis et al., 2008). The sensors monitor the electric signal of the cells interacting with viruses making their identification possible. ANNs were trained with the responses of the biosensors to obtain a classification model of the culture cells infected. ANNs based modelling approaches have also been applied in cell culture practice (Prasad & Dutta Gupta, 2008). One of the most important topics in in vitro culture is related with the supply of carbon source, commonly sucrose, to determine the effect on plant growth and physiological parameters. Tani and co-workers (1992) developed a growth model for in vitro shoots of alfalfa describing the effects of CO2 inside the culture vessel and the sucrose content. This model increases the understanding of in vitro processes in a non-deterministic way. Three years later, a comparison between the deterministic mathematical model Extended Kalman Filter approach and ANNs was performed (Albiol et al., 1995). Authors stated the usefulness of neural networks for modelling at less cost, time and a smaller dataset. Pattern recognition and classification models are commonly applied in plant tissue culture studies (Prasad & Dutta Gupta, 2008). Usually, the selection of embryos inside the embriogenic cultures is laborious, cost intensive and time-consuming. An image analysis pattern recognition system was developed by Zhang and coworkers (1999) using ANNs to select embryos of Douglas fir: the contour of embryo images was segmented, digitalized and converted into numerical values after the discrete and fast Fourier transformation (values obtained were higher than 80% for normal embryos). In another study, embryos of sugarcane from callogenic culture were selected by image analysis using machine vision analysis (MVA) confirming the technique as a rapid, non-invasive method for qualitative evaluation and quantification of in vitro regenerated plantlets (Honda et al., 1999). Finally, regenerated

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

205

plantlets could be sorted using their photometrical behaviour of their leaves in Red, Blue and Green colour regimes combining image analysis and ANNs (Mahendra et al., 2004). Transference to field the micropropagated plantlets is a typical bottleneck in micropropagation protocols. Optimizing rhizogenesis and subsequent acclimatization is highly recommended to improve efficient propagation yields (Gago, 2009). The quality and the functionality of the root system, the vigour of the plantlets and other physiological status will be responsible for the quality of the plantlets in ex vitro conditions (Gago et al., 2009). Recently (Gago et al., 2010b), ANNs were used to model in vitro rhizogenesis and subsequent acclimatization data simultaneously of grapevine Vitis vinifera L. cv. Albariño. Studied inputs/factors (cultivar, IBA concentration and exposure time to IBA) showed significant effects on root number, ex vitro leaves, number of nodes and height of the acclimatized plantlets, the exposure time to the synthetic auxin IBA being the more relevant. The model allowed optimal predictions for every studied cultivar. The knowledge derived through ANNs can be easily increased by training the model by adding to the database new inputs (salt concentration, type of medium, other plant hormone, etc.) and/or outputs (plantlets weight, chlorophyll and carotenes content, stomata analysis, etc.).

5. Genetic algorithms
Once an ANN model has been obtained it is easy to predict what will be the output for a specific set of inputs, or in other word to formulate “what if” questions obtaining accurate responses (Fig. 3). This consultation mode will provide insight into the process studied. However sometimes, the main research objective is to determine the combination of input parameters that will provide the optimum result, that is, an optimization process which means to formulate “how to get” questions on the best/highest...output. In these cases a different AI technique, the genetic algorithms, can be applied.

Fig. 3. The relationship between modeling and optimization from an ANN model (Modified from Rowe and Roberts, 2005). “The genetic algorithm is an optimization technique based on evolutionary principles” (Cartwright, 1993). Genetic algorithms are based on the biological principles of genetic variation and natural selection, mimicking the basic ideas of evolution over generations. As Rowe & Roberts declare: “An optimization process evolves finding the best solution for a specific problem” (Rowe & Roberts, 1998). Genetic algorithm randomly generates a set of

206

Artificial Neural Networks - Methodological Advances and Biomedical Applications

candidate solutions to the problem. Solutions from one population are selected according to their fitness of evolving new populations to the problem. This is repeated until some condition is satisfied. After several generations an optimum must be achieved because the most suitable becomes the solution and therefore the more chances it has to reproduce. For a genetic algorithm to function it must possess several features. Firstly, a numerical description of how good a solution is to the problem. Secondly, a logical method of selecting individual solutions to become parents of the next generation must be fixed. And finally, a logical method of mixing the different elements to produce new solutions is necessary (Mitchell, 1998; Glezakos et al., 2010). 5.1. Applications of genetic algorithms to plant biology Genetic algorithms have been used in plant science for different optimization processes. Noguchi & Terao (1997) have developed a mobile robot for harvesting fruit automatically and genetic algorithms were designed to find the optimal space solution for path planning. Hybrid systems combining neural networks and genetic algorithms have also been used for optimizing the quality of fruits stored under a controlled environment (Morimoto et al., 1997; Morimoto & Hashimoto, 2000) and for plant virus identification through a Bio-Electric Recognition Assay (BERA) (Glezakos et al., 2010). More complex in vitro culture processes such as shoot proliferation, root formation (rhizogenesis) and plantlets acclimatization have been modeled by ANNs and successfully optimized by genetic algorithms in woody fruit plants, such as kiwifruit (Gago et al., 2010a) and grapevine (Gago et al., 2010b).

6. Neurofuzzy logic
Human knowledge is typically built on linguistic tags (characterized by uncertainty or imprecision) and not on quantitative mathematical data. Many times words have higher significance in the real world than a collection of numerical data (Fig. 4) the basis being to solve problems, make decisions or draw conclusions.

Fig. 4. Picture on the importance of precision and significance in the real world of plant research.

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

207

Neurofuzzy logic is a hybrid system technology developed to break in this topic. It combines the adaptive learning capabilities from ANNs with the generality of representation from fuzzy logic (Shao et al., 2006). Fuzzy logic is an AI tool especially useful in problem solving. Fuzzy set theory was introduced by Zadeh (1965) as an extension of the classical set theory, which enables the processing of imprecise information using the membership concept (Adriaenssens et al., 2004). Prof. Zadeh illustrated the concept of fuzzy set and the degree of membership with the classical “tall man” example (Fig. 5). The conventional characteristic mapping of a classical logic set determines that a man is tall when his height is over 1.80 m. Zadeh extended the traditional definition of a logic premise from having just two extremes (either a man is over 1.80 m, so he is tall or lower 1.80 m, so he is not tall) to one in which there is a range in degree of truth from 0 to 1. For example new sets can be described qualitatively by terms as very low, low, average, tall or very tall. However, following the classical logic, a small difference of just 2 cm from 1.79 to 1.81 m, induces classification of the man in two completely different categories. This does not seem really “logic”. But, using the fuzzy set theory, an element of those sets can be assigned to a fuzzy set with its membership degree ranging from zero to one, so two men 1.79 m and 1.81 m tall belong to fuzzy set “tall man” with membership degrees of 0.70 and 0.90 respectively (Fig. 5).

Fig. 5. Comparison between classical set theory and fuzzy set theory to illustrate the Zadeh´s example of the “tall man” (Modified from Zadeh, 1965). These kinds of fuzzy data or fuzzy variables can be numerically characterized, but a fuzzification process is necessary. For better understanding of this concept, we will choose a typical tissue culture proliferation experiment in order to evaluate the effect of light intensity on the success of the proliferation of kiwifruit shoots. Light conditions for an in vitro culture experiment can be expressed by the Photosynthetic Photon Flux Density (PPFD) parameter (Fig. 6). The x axis is the PPFD with ranges for the fuzzy sets low, medium or high. The y axis represents the membership function and ranges 0 to 1 (also could be expressed from 0 to 100 per cent). It can be seen that a PPDF light of 80 mmol-2s-1 can be regarded as both low and medium PPFD with membership functions of 0.7 and 0.4 respectively. In other words 80 mmol-2s-1 is low light in a greater degree than it is medium light. Therefore, fuzzy logic sets labels qualitatively using linguistic terms and also assigns varying degrees of membership called membership functions. The membership function

208

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Fig. 6. Examples of fuzzy sets for light intensity (PPFD) for an in vitro plant tissue culture experiment. then is subjective in nature and is a matter of definition rather than measurement. This process allows the interaction between linguistic terms (low, medium, high) and the membership functions making the terms meaningful to a computer. Additionally it makes expressing the behaviour of a system possible using natural language and enhancing the possibility of concise description of complex tasks or process. In fuzzy-rule based systems, after modelling, knowledge is presented by IF-THEN rules. Fuzzy rules consist of two parts: an antecedent part stating conditions on the input variable(s), and a consequent part describing the corresponding values of the output variable(s). Given particularly values of the input variables, the degree of fulfilment of each rule is obtained by aggregating the membership degrees of these input values into the respective fuzzy sets. Going back to the example described previously if an evaluation of the effect of light conditions on the length of the plants in an in vitro culture experiment were carried out, the IF THEN rules could be similar to those presented in Table 1: IF PPFD is low THEN the plant length obtained is HIGH with a membership of 89% (more detailed information can be obtained in Gago et al., 2010d). IF PPFD is LOW IF PPFD is MID IF PPFD is HIGH IF...THEN RULES THEN Plant length is THEN Plant length is THEN Plant length is HIGH (0.89) LOW (0.76) LOW (0.68)

Table 1. Examples of a fuzzy output using IF THEN rules describing the effect of the light intensity (PPFD) on the plant length in an in vitro culture experiment. The fuzzy output is determined by the degrees of fulfilment and the consequent parts of the rules (Adriaenssens et al., 2004). The logical structure of rules facilitates the comprehension of a semi-qualitative manner, similar to that used by the human brain to analyze the real world (Babuska, 1998). The major capabilities of fuzzy logic are the flexibility, the tolerance with uncertainty and vagueness and the possibility of modelling non linear functions, searching for consistent

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

209

patterns or systemic relationships between variables in a complex dataset, data mining and promoting deep understanding of the processes studied by generating comprehensible and reusable knowledge in an explicitly format (Setness et al., 1998; Shao et al., 2006; Yuan et al., 2008; Landin et al., 2009; Gago et al., 2010d). Chen & Mynett (2003) have argued that definition of membership functions and induction of inference rules is the most difficult part in a fuzzy logic process. Fuzzy logic can be combined with neural networks to produce neuro-fuzzy techniques. Those hybrid systems combine the generality and flexibility of representation, a feature of fuzzy logic, with the powerful learning and adaptive capability of neural networks (Babuska, 1998; Adriaenssens et al., 2004). 6.1 Applications of neurofuzzy logic to plant biology There are not many works in the literature on neurofuzzy logic in plant science. The neurofuzzy logic technology was used for controlling and modelling Ginjo sake brewing process: the interaction between sensory evaluation and the chemical composition of sake, beer and coffee was studied, showing this technology to have a high level of accuracy (Hanai et al., 1997). Also, there are some works related to the monitoring of herbicide sprayed in cornfields with a system that includes real-time image processing, weed identification, mapping of weed density, and sprayer control using a digital camera. Simulations using different fuzzy rules and membership functions indicated that the precision spraying has potential for reducing water pollution from herbicides needed for weed control in a corn field (Yang et al., 2003). Finally, hybrid systems as NUFZY involving a fuzzy approach and the training algorithm OLS (orthogonal least squares) has also been used to model accurately the lettuce growth and the greenhouse temperature (Tien & Van Straten, 1998). Recently, Gago and coworkers (Gago et al., 2010d) compared the utility of the traditional statistical analysis and neurofuzzy logic technology for dataset highly complex and with great variability of direct rooting and subsequent acclimatization of grapevine. Neurofuzzy logic showed higher accuracy to identify the interaction effects between the factors: type of auxin (IBA, IAA and NAA), auxin concentration (1 to 50 mM) and sucrose concentration (0 to 9%) than conventional statistical analysis. Also, neurofuzzy showed a considerable potential for data mining and retrieve knowledge from the complex dataset. Understanding was increased thanks to IF-THEN rules generated from the model to facilitate researchers interpretation of the results, main effects and their consequences. Considerable efforts have been made to understand artificial neural networks and neurofuzzy logic capacities in this sense, and many works are expected, in the near future, to provide a comprehensive insight into the expediency of processing networks in interpreting the database derived from plant biology research. In addition, one of the major advantages of these hybrid techniques is the capacity to model and estimate different complex processes.

7. Future perspectives
Biological systems are complex to understand. They have different scales of biological organization (genetic, biochemical, physiological...) and different factors influence them. Nowadays modern technology gives us the opportunity to generate a huge amount of biological data (Brady & Provart, 2009). This storm of information would be useless if at the same time the technology do not solve the problems associated of analysing, integrating and

210

Artificial Neural Networks - Methodological Advances and Biomedical Applications

extracting knowledge from those data. If the technology creates a problem, the technology should solve it. AI technologies, the ones in use and the ones coming out in the future, will help the researchers with those difficult tasks: to integrate variable information, to explain what is going on, to model and finally to predict what will happen in a specific situation. As described above neural networks can be used for a wide range of application domains in the general area of plant biology, and gives models as least as good as those obtained using statistical modelling. Neural networks combined with genetic algorithms can predict the combination of variables that would yield optimum solution when independent variables are fed into the network as it has been pointed out in this review. Neural network technologies have also spectacular advantages over other systems as the ability to capture non-linear relationships in the data (wherever their origin or type and even from incomplete data sets), without requiring prior knowledge from the user. In fact, the user does not need to have a deep mathematical or statistical background to employ effectively neural systems. We think that once having overcome the natural reluctance of scientists to these new technologies, they will impose as the usual way for dealing with biological results. For that purpose, we strongly recommend the use of software packages, which incorporates visualization and data manipulation capabilities, within an easy to use interface, so the users do not need to be experts in neural computer (ie those used in Gago et al., 2010a, d). Finally, the knowledge derived through neural networks can be easily increased by adding new data (inputs and/or outputs) to the database, giving new insight to understand the regulatory process controlling developmental and physiological processes in plants, as a whole. The knowledge obtained in this way should be crucial for both basic and applied plant biology.

8. Acknowledgments
Special thanks to Ms. J. Menis for her help in the correction of the English version of the work, Prof. Ray Rowe, Dr. Elizabeth Colbourn and Prof. Sjef Smeekens provided useful comments and critical review to this manuscript. This work was supported by Regional Government of Xunta de Galicia; exp.2007/097 and PGIDIT02BTF30102PR. PPG (PR2010-0357) and ML (PR2010-0460) thanks to and Minister of Education of Spain for partially funding the sabbatical year at Faculty of Science, University of Utrecht, Netherlands.

9. References
Adriaenssens, V.; De Baets, B.; Goethals, P.L.M. & De Pauw, N. (2004). Fuzzy rule-based methods for decision support in ecosystem management. The Science of Total Environment, 319, 1-12. Achanta, A.S.; Kowalski, J.G. & Rhodes, C.T. (1995). Artificial neural network: implications for pharmaceutical sciences. Drug Development and Industrial Pharmacy, 21, 119-155. Albiol, J.; Campmajo, C.; Casas, C. & Poch, M. (1995).Biomasa estimation in plant cell cultures: A neural network approach. Biotechnology Progress, 11, 8-92. Alvarez-Buylla, E.R.; Benitez, M.; Davila, E.B.; Chaos, A.; Espinosa-Soto, C. & Padilla Longoria P. (2007). Gene regulatory network models for plant development. Current Opinion in Plant Biology, 10, 83–91. Anderson, P.E. & Jensen, H.J. (2005). Networks properties, species abundance and evolution in a model of evolutionary ecology. Journal of Theoretical Biology, 232, 4, 551-558.

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

211

Babuska, R. (1998). Fuzzy modelling for control, In: International Series in Intelligent Technologies, Babuska R. (Ed.), 1-8, Kluwer Academic Publishers, Massachussets, USA. Bar-Joseph, Z.; Gerber, G.K.; Lee, T.I.; Rinaldi, N.J.; Yoo, J.Y.; Robert, F.; Gordon, D.B.; Fraenkel, E.; Jaakkola, T.S.; Young, R.A. & Gifford, D.K. (2003). Computational discovey of genes modules and regulatory networks. Nature Biotechnology, 21, 11, 1337-1342. Belostotsky, D.A. & Rose, A.B. (2005). Plant gene expression in the age of systems biology: integrating transcriptional and posttranscriptional events. Trends in Plant Science, 10, 347–353. Boone, C.; Bussey, H. & Andrews, B.J. (2007). Exploring genetic interactions and networks with yeast. Nature Reviews Genetics, 8, 6, 437–449. Brady, S.M. & Provart, N.J. (2009). Web-queryable large-scale data sets for hypothesis generation in plant biology. Plant Cell 21, 1034-1051. Brauer, F. & Castillo-Chavez, C. (2001). Mathematical models in population biology and epidemiology (Texts in Applied Mathematics 40), Springer Verlag, Berlin. Cartwright, H.M. (1993). Applications of Artificial Intelligence in Chemistry, Oxford University Press, Oxford. Chen, Q. & Mynett A.E. (2003). Integration of data mining techniques and heuristic knowledge in fuzzy logic modelling of eutrophication in Taihu Lake. Ecological Modelling, 162, 1-2, 55-67. Colbourn, E. (2003). Neural computing: enable intelligent formulations. Pharmaceutical Technology Supplement, 16-20. Colbourn, E. & Rowe, R.C. (2005). Neural computing and pharmaceutical formulation. In: Encyclopaedia of pharmaceutical technology, Swarbrick, J. & Boylan, J.C. (Eds), Marcel Dekker, New York. Coruzzi, G.M.; Burga, A.R.; Katari, M.S. & Gutiérrez, R.A. (2009). Systems biology: principles and applications in plant research, In: Annual Plant Reviews, Vol. 35, Plant Systems Biology, Gloria M. Coruzzi and Rodrigo A. Gutiérrez (Eds.), 3-40, WileyBlackwell, United Kingdom. De Vries, G.; Hillen, T.; Lewis, M.; Müller, J. & Schönfisch, B. (2006). A course in mathematical biology: quantitative modelling with mathematical and computational methods. SIAM, Society for Industrial and Applied Mathematics, Philadelphia, PA. Eerikäinen, T.; Linko, P.; Linko, S.; Siimes, T. & Zhu Y.H. (1993). Fussy logic and neural networks applications in food science and technology. Trends in Food Science and Technology, 4, 237-242. Frossyniotis, D.; Anthopoulos, Y.; Kintzios, S.; Moschopoulou, G. & Yialouris, C.P. (2008). Artificial neural network selection for the detection of plant viruses. World Journal of Agricultural Sciences, 4, 1, 114-120. Fukuda, T.; Shiotani, S.; Aral, F.; Takeuchi, N.; Sasaki, K. & Kimoshita, T. (1991). Cell recognition by image processing (1st report, recognition of dead or alive plant cells by neural network). Nihon Kikaigakkai Ronbunshu, 57, 77-84. Gago, J. (2009). Biotecnología de Vitis vinifera L.: Modelización mediante Inteligencia Artificial. Doctoral Thesis, Universidade de Vigo, Vigo, Spain.

212

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Gago, J.; Martínez-Núñez, L.; Molist, P. & Gallego, P.P. (2009). Effect of three auxins on in vitro rooting in soil and acclimatization of Vitis vinifera L. Albariño. Acta Horticulturae, 812, 359-364. Gago, J.; Martínez-Núñez, L.; Landín, M. & Gallego, P.P. (2010a). Artificial neural networks as an alternative to the traditional statistical methodology in plant research. Journal of Plant Physiology, 167, 23-27. Gago, J.; Landín, M. & Gallego, P.P. (2010b). Artificial neural networks modeling the in vitro rhizogenesis and acclimatization of Vitis vinifera L. Journal of Plant Physiology, 167, 1226-1231. Gago, J.; Martínez-Núñez, L.; Landín, M. & Gallego, P.P. (2010c). Strengths of artificial neural networks in modelling complex plant processes. Plant Signaling and Behavior 5, 6, 1-3. Gago, J.; Landín, M. & Gallego, P.P. (2010d). A neurofuzzy logic approach for modeling plant processes: a practical case of in vitro direct rooting and acclimatization of Vitis vinifera L. Plant Science, 179, 241-249. Glezakos, T.J.; Moschopoulou, G.; Tsiligiridis, T.A.; Kintzios, S. & Yialouris, C.P. (2010). Plant virus identification based on neural networks with evolutionary preprocessing. Computers and Electronics in Agriculture, 70, 263-275. Green, T.R.; Salas J.D.; Martinez, A. & Erskine R.H. (2007). Relating crop yield to topographic attributes using Spatial Analysis Neural Networks and regression. Geoderma, 23-37. Guégan, J.F.; Lek, S. & Oberdoff T. (1998). Energy availability and habitat heterogeneity predict global riverine fish diversity. Nature, 391, 382-384. Hammer, G.L.; Kropff, M.J.; Sinclair, T.R. & Porter, J.R. (2002). Future contributions of crop modelling: from heuristics and supporting decision making to understanding genetic regulation and aiding crop improvement. European Journal of Agronomy, 18, 15-31. Hammer, G.L.; Sinclair, T.R.; Chapman, S.C. & van Oosterom E. (2004). On systems thinking, systems biology, and the in silico plant. Plant Physiology, 134, 909-911. Hanai, T.; Katayama, A.; Honda, H. & Kobayaski, T. (1997). Automatic fuzzy modelling for Ginjo sake brewing process usingfuzzy neural network. Journal of Chemical Engineering of Japan, 30, 94-100. Hilbert, D.W. & Ostendorf B. (2001). The utility of neural networks for modelling the distribution of vegetation in past, present and future climates. Ecological modelling, 146, 311-327. Hirai, M.Y.; Yano, M.; Goodenowe, D.B.; Kanaya, S.; Kimura, T.; Awazuhara, M.; Arita, M.; Fujiwara, T. & Saito, K. (2004). Integration of transcriptomics and metabolomics for understanding of global responses to nutritional stresses in Arabidopsis thaliana. Proceedings of The National Academy of Sciences USA, 101, 10205–10210. Honda, H.; Takikawa, N.; Noguchi, H.; Hanai, T. & Kobayashi, T. (1997). Image analysis associated with fuzzy neural network and estimation of shoot length of regenerated rice callus. Journal of Fermentation and Bioengineering, 84, 342-347. Honda, H.; Ito, T.; Yamada, J.; Hanai, T. & Matsuoka, M. (1999). Selection of embryogenic sugarcane callus by image analysis. Journal of Bioscience and Bioengineering, 87, 5, 700-702. Huang, Y. (2009). Advances in artificial neural networks - Methodological development and application. Algorithms, 2, 973-1007.

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

213

Hudson, D.L. & Cohen, M.E. (2000). Neural networks and artificial intelligence for biomedical engineering. IEEE Press, New York. Hussain, B. & Kabuka, M.R. (1998). A novel feature recognition neural network and its application to character recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 98-106. Jaillon, O; Aury, J.M.; Noel, B.; Policriti, A.; Clepet, C.; Casagrande, A.; Choisne, N.; Aubourg, S.; Vitulo, N.; Jubin, C.; Vezzi, A.; Legeai, F.; Hugueney, P.; Dasilva, C.; Horner, D.; Mica, E.; Jublot, D.; Poulain, J.; Bruyere, C.; Billault, A.; Segurens, B.; Gouyvenoux, M.; Ugarte, E.; Cattonaro, F.; Anthouard, V.; Vico, V.; Del Fabbro, C.; Alaux, M.; Di Gaspero, G.; Dumas, V.; Felice, N.; Paillard, S.; Juman, I.; Moreoldo, M.; Scalabrin, S.; Canaguier, A.; Le Clainche, I.; Malacrida, G.; Durand, E.; Pesole, G.; Laucou, V.; Chatelet, P.; Merdinoglu, D.; Delledonne, M.; Pezzotti, M.; Lecharny, A.; Scarpelli, C.; Artiguenave, F.; Pe, M.E.; Valle, G.; Morgante, M.; Caboche, M.; Adam-Blodon, A.F.; Weissenbach, J.; Quetier, F. & Wincker, P. (2007). The grapevine genome sequence suggests ancestral hexaploidization in major angiosperm phyla. Nature, 449, 463-467. Jiménez, D.; Pérez-Uribe, A.; Satizábal, H.; Barreto, M.; Van Damme, P. & Marco, T. (2008). A Survey of Artificial Neural Network-Based Modeling in Agroecology. In: Soft Computing applications in industry, STUDFUZZ. B. Prasad (Ed), 247-269 SpringerVerlag, Berlin-Heidelberg Germany. Karim, M.N.; Yoshida, T.; Rivera, S.L.; Saucedo, V.; Eikens, B. & Oh, G. (1997). Global and local neural network models in biotechnology: application to different cultivation processes. Journal of Fermentation and Bioengineering, 83, 1, 1-11. Katagiri, F. (2003). Attacking complex problems with the power of systems biology. Plant Physiology, 132, 417–419. Kaul, M.; Hill, R.L. & Walthall, C. (2005). Artificial neural networks for corn and soybean yield prediction. Agricultural Systems, 85, 1-18. Khazaei, J.; Naghavi, M.R.; Jahansouz, M.R. & Salimi-Khorshidi, G. (2008). Yield estimation and clustering of chickpea genotypes using soft computing techniques. Agronomy Journal, 100, 1077-1087. Kehagias, A.; Panagiotou, N.; Maslaris, V.; Petridis, V. & Petrou, L. (1998). Predictive modular neural networks methods for prediction of sugar beet crop yield. In: IFAC Conference on Control Applications and Ergonomics in Agriculture, Athens, Greece, pp 1-5. Kirschner, M.W. (2005). The meaning of systems biology. Cell, 121, 503–504. Kitano, H. (2002). Systems biology: a brief overview. Science, 295, 1662-1664. Kliebenstein, D.J. (2010). Systems biology uncovers the foundation of natural genetic diversity. Plant Physiology, 152, 480-486. Kovalenko, I.B. & Riznichenko, G.Y. (2007). Multiparticle direct simulation of photosynthetic electron transport processes. In: Mathematical Modelling of Biological Systems, Volume I, Deutsch A, Brush L, Byrne H, de Vries G & Herzel H (Eds), 3-9, Birkhäuser, Boston. Landín, M.; Rowe, R.C. & York, P. (2009). Advantages of neurofuzzy logic against conventional experimental design and statistical analysis in studying and developing direct compression formulations. European Journal of Pharmaceutical Science, 38, 325-331.

214

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Legg, S. & Hutter, M. (2007). Universal Intelligence: A Definition of Machine Intelligence. Minds and Machines, 17, 391–444. Lek, S. & Guégan, J.F. (1999). Artificial neural networks as a tool in Ecological Modelling, an introduction. Ecological Modelling, 120, 65-73. Lennox, B.; Montague, G.A.; Frith, A.M.; Gent, C. & Bevan, V. (2001). Industrial application of neural networks – an investigation. Journal of Process Control, 11, 497-507. Ma, L. and Khorasani, K. (2004). Facial expression recognition using constructive feedforward neural networks. IEEE Transactions on Systems Man and Cybernetics part B-CUverbetics by Cybernetics, 34, 3, 1588-1595. Mahendra, V.; Prasad, V.S.S. & Dutta Gupta, S. (2004). Trichromatic sorting of in vitro regenerated plants of gladiolus using adaptive resonance theory. Current Science, 87, 348-353. Mehrota, S.; Prakash, O.; Mishra, B.N. & Dwevedi, B. (2008). Efficiency of neural networks for prediction of in vitro culture conditions and inoculums properties for optimum productivity. Plant Cell Tissue and Organ Culture, 95, 29-35. Ming, R.; Hou, S.; Feng, Y.; Yu, Q.; Dionne-Laporte, A.; Saw, J. H.; Wang, W.; Ly, B. V.; Lewis, K. L. T.; Salzberg, S. L.; Feng, L.; Jones, M. R.; Skelton, R. L.; Murray, J.E.; Chen, C.; Qian, W.; Shen, J.; Du, P.; Eustice, M.; Tong, E.; Tang, H.; Lyons, E.; Paull, R.E.; Michael, T.P.; Wall, K.; Rice, D.W.; Albert, H.; Wang, M.; Zhu, Y.J.; Schatz, M.; Nagarajan, N.; Acob, R.A.; Guan, P.; Blas, A.; Wai, C.M.; Ackerman, C.M.; Ren, Y.; Liu, C.; Wang, J.; Wang, J.; Na, J.; Shakirov, E.V.; Haas, B.; Thimmapuram, J.; Nelson, D.; Wang, X.; Bowers, J.E.; Gschwend, A.R.; Delcher, A.L.; Singh, R.; Suzuki, J.Y.; Tripathi, S.; Neupane, K.; Wei, H.; Irikura, B.; Paidi, M.; Jiang, N.; Zhang, W.; Presting, G.; Windsor, A.; Navajas-Perez, R.; Torres, M.; Feltus, F.A.; Porter, B.; Li, Y.; Burroughs, A.M.; Luo, M.; Liu, L.; Christopher, D.A.; Mount, S.M.; Moore, P.H.; Sugimura, T.; Jiang, J.; Schuler, M.A.; Friedman, V.; Mitchell-Olds, T.; Shippen, D.E.; dePamphilis, C.W.; Palmer, J.D.; Freeling, M.; Paterson, A.H.; Gonsalves, D.; Wang, L. & Alam, M. (2008). The draft genome of the transgenic tropical fruit tree papaya (Carica papaya Linnaeus). Nature, 452, 991-996. Mitchell, M. (1998). An introduction to genetic algorithms, MIT Press, Cambridge, USA. Mize, C.W.; Kenneth, J.K. & Compton, M.E. (1999). Statistical considerations for in vitro research: II – data to presentation. In Vitro Cell & Developmental Biology – Plant, 35, 122-126. Morimoto, T.; De Baerdemaeker, J. & Hashimoto, Y. (1997). An intelligent approach for optimal control of fruit-storage process using neural networks and genetic algorithms. Computers and Electronics in Agriculture, 18, 205-224. Morimoto, T. & Hashimoto, Y. (2000). An intelligent control for greenhouse automation, oriented by the concepts of SPA and SFA-an application to a post-harvest process. Computers and electronics in agriculture, 29, 3-20. Müller, B.; Reinhardt, J. & Strickland, M.T. (1995). Neural Networks. An Introduction. Springer, Berlín. Nemhauser, J.L.; Mallof, J.N. & Chory, J. (2003). Building integrated models of plant growth and development. Plant Physiology, 132, 2, 436-439. Nijhout, H.F.; Maini, P.K.; Madzvamuse, A.; Wathen, J.W. & Sekimura, T. (2003). Pigmentation pattern formation in butterflies: experiments and models. Comptes Rendus Biologies, 326, 8, 717-727.

Artificial Neural Networks Technology to Model and Predict Plant Biology Process

215

Noguchi, N. & Terao, H. (1997). Path planning of an agricultural mobile robot by neural network and genetic algorithm. Computers and Electronics in Agriculture, 18, 187-204. Ondimu, S.N. & Murase, H. (2008). Comparison of plant water stress detection ability of color and graylevel texture in sunagoke moss. Transactions of the ASABE, 51, 3, 11111120. Plumb, A.P.; Rowe, R.C.; York, P. & Brown, M. (2005). Optimisation of the predictive ability of artificial neural network (ANN) models: a comparison of three programs and four classes of training algorithm. European Journal of Pharmaceutical Science, 25, 395-405. Prasad, V.S.S. & Dutta Gupta, S. (2008). Applications and potentials of artificial neural networks in plant tissue culture. In: Plant Tissue Culture Engineering, Dutta Gupta, S. & Ibaraki Y. (Eds), 47-67, Springer-Verlag: Berlin, Germany. Rowe, R.C. & Roberts, R.J. (1998). Artificial intelligence in pharmaceutical product formulation: neural computing and emerging technologies. Pharmaceutical Science & Technology Today, 1, 5, 200-205. Rowe, R.C. & Roberts, R.J. (2005). Intelligent Software for Product Formulation, Taylor & Francis, London. Rumelhart, D.E. & McClelland, J.L. (1986). Parallel Distributed Processing: Explorations in the Microstructures of Cognition, MIT Press, Cambridge, USA. Russell, S.J. & Norvig, P. (2003). Artificial Intelligence: A Modern Approach (2nd ed.), Upper Saddle River, NJ: Prentice Hall, USA. Schauer, N. & Fernie, A.R. (2006). Plant metabolomics: towards biological function and mechanism. Trends in Plant Science, 11, 508–516. Setnes, M.; Babuska, R. & Verbruggen, H.B. (1998). Rule-based modelling: precision and transparency. IEEE Transactions on Systems Man and Cybernet Part C-Applications and Reviews, 28, 1, 165-169. Shao, Q.; Rowe, R.C. & York, P. (2006). Comparison of neurofuzzy logic and neural networks in modelling experimental data of an immediate release tablet formulation. European Journal of Pharmaceutical Science, 28, 394-404. Shao, Q.; Rowe, R.C. & York, P. (2007). Comparison of neurofuzzy logic and decision trees in discovering knowledge from experimental data of an inmediate release tablet formulation. European Journal of Pharmaceutical Science, 31, 129-136. Schuster, S.; Knoke, B. & Marhl, M. (2005). Differential regulation of proteins by bursting calcium oscillations – a theoretical study. BioSystems, 81, 49-63. Stollenwerk, N.; Drepper, F. & Siegel, H. (2001). Testing nonlinear stochastic models on phytoplankton biomass time series. Ecological Modelling, 144, 261-277. Struik, P.C.; Yin, X. & de Visser, P. (2005). Complex quality traits: now time to model. Trends in Plant Science, 10, 513-516. Sweetlove, L.J.; Last, R.L. & Fernie, A.R. (2003). Predictive metabolic engineering: a goal for systems biology. Plant Physiology, 132, 2, 420–425. Takayama, K.; Fujikawa, M. & Nagai, T. (1999). Artificial neural network as a novel method to optimize pharmaceutical formulations. Pharmaceutical Research, 16, 1, 1-6. Tani, A.; Murase, H.; Kiyota, M. & Honami, N. (1992). Growth simulation of alfalfa cuttings in vitro by kalman filter neural network. Acta Horticulturae, 319, 671-676. Tardieu, F. (2003). Virtual plants: modelling as a tool for the genomics of tolerance water deficit. Trends in Plant Science, 8, 1, 9-14.

216

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Taylor, J.G. (1996). Neural networks and their applications. Taylor J. G. (Eds) John Wiley & Sons Ltd. England. Thai, C.N. & Shewfelt, R.L. (1991). Modelling sensory colour quality of tomato and peach: neural networks and statistical regression. Transaction of the ASAE, 34, 3, 950-955. Thum, K.E., Shasha, D.E.; Lejav, L.V. & Coruzzi, G.M. (2003). Light- and carbon-signaling pathways. Modeling circuits of interactions. Plant Physiology, 132, 440–452. Tien, B.T. & Van Staten, G. (1998). A neurofuzzy approach to identify lettuce growth and greenhouse climate. Artificial Intelligence Review, 12, 71-93. Tracqui, P. (2006). Mechanical instabilities as a central issue for insilico analysis of cell dynamics. Proceedings of the IEEE, 94, 4, 710-724. Trewavas, A. (2006). A brief history of systems biology. ‘Every object that biology studies is a system of systems.’ Francois Jacob (1974). Plant Cell 18, 2420–2430. Uozumi, N.; Yoshino, T.; Shiotani, S.; Suehara, K.; Arai, F.; Fukuda, T. & Kobayashi, T. (1993). Application of image analysis with neural network for plant somatic embryo culture. Journal of Fermentation and Bioengineering, 76, 505-509. Wang, E.; Robertson, M.J.; Hammer, G.L.; Carberry, P.S.; Holzworth, D.; Meinke, H.; Chapman, S.C.; Hargreaves, J.N.G.; Huth, N.I. & McLean, G. (2002). Development of a generic crop model template in the cropping system model APSIM. European Journal of Agronomy, 18, 121-140. Welch, S.M.; Roe, J.L. & Dong, Z. (2003). A genetic neural network model of flowering time control in Arabidopsis thaliana. Agronomy Journal, 95, 71-81. Whittaker, A.D.; Park, B.P.; McCauley, J.D. & Huang Y. (1991). Ultrasonic signal classification for beef quality grading through neural networks. In: Proceedings of automated agriculture for the first 21st century, Chicago, IL, USA, pp 116-125. Williams, J.C.; ReVelle, C.S. & Levin, S.A. (2004). Using mathematical optimization models to design nature reserves. Frontiers in Ecology and the Environment, 2, 98-105. Yang, C.C. (2003). Development of a herbicide application map using artificial neural networks and fuzzy logic. Agricultural Systems, 76, 561-574. Yuan, J.S.; Galbraith, D.W.; Dai, S.Y.; Griffin, P. & Stewart, N. Jr. (2008). Plant systems biology comes of age. Trends in Plant Science, 13, 4, 165-171. Zadeh, L. (1965). Fuzzy sets. Information and Control, 8, 338-353. Zhang, C.; Timmis, R. & Hu, W.S. (1999). A neural network based pattern recognition system for somatic embryos of Douglas fir. Plant Cell Tissue and Organ Culture, 56, 1, 25-35. Zupan, J. & Gasteiger, J. (1993). Neural Networks for chemists – an introduction. VCH Publishers, New York.

11
The Usefulness of Artificial Neural Networks in Predicting the Outcome of Hematopoietic Stem Cell Transplantation
1Division

Giovanni Caocci1, Roberto Baccoli2 and Giorgio La Nasa1

of Hematology and Hematopoietic Stem Cell Transplantation, Department of Internal Medical Sciences, University of Cagliari 2Technical Physics Division, Faculty of Engineering, Department of Engineering of the Territory, University of Cagliari Italy

1. Introduction
1.1 Artificial neural networks in clinical medicine In Medicine, several tools have been developed for the prediction of clinical outcomes following drug treatment and other medical interventions. The standard approach for a binary outcome is to use logistic regression (LR), however, this method requires formal training and a profound knowledge of statistics (Royston, 2000; Harrel et al., 1996). LR is used to predict a categorical (usually dichotomous) variable from e set of predictor variables; it has been especially popular with medical research in which the dependent variable is whether or not a patient has a disease. Over the past years, artificial neural networks (ANNs) have increasingly been used as an alternative to LR analysis for prognostic and diagnostic classification in clinical medicine (Schwarzer et al., 2000). ANNs are composed of simple elements operating in parallel inspired by biological nervous systems. As in nature, the network function is determined largely by the connections between elements. After training with retrospective data ANNs are capable of making intelligent predictions given new, limited information. The growing interest in ANNs has mainly been triggered by their ability to mimic the learning processes of the human brain. However, the issue remains as to how these ANNs actually succeed in recognizing patterns within data that are too complex for the human brain. From here derives the so-called “black-box” aspect of ANNs. The network operates in a feed-forward mode from the input layer through the hidden layers (like in a black box) to the output layer. Exactly what interactions are modeled in the hidden layers is still a knot that remains untied. Each layer within the network is made up of computing nodes with remarkable data processing abilities. Each node is connected to other nodes of a previous layer through adaptable inter-neuron connection strengths known as synaptic weights. ANNs are trained for specific applications, such as pattern recognition or data classification, through a learning process and knowledge is usually retained as a set of connection weights. The backpropagation algorithm and its variants are learning algorithms that are widely used in neural networks. With backpropagation, the input data is repeatedly presented to the

218

Artificial Neural Networks - Methodological Advances and Biomedical Applications

network. Each time, the output is compared to the desired output and an error is computed. The error is then fed back through the network and used to adjust the weights in such a way that with each iteration it gradually declines until the neural model produces the desired output. ANNs have been successfully applied in the fields of mathematics, engineering, medicine, economics, meteorology, psychology, neurology, and many others. Indeed, in medicine, they offer a tantalizing alternative to multivariate analysis, although their role remains advisory since no convincing evidence of any real progress in clinical prognosis has yet been produced (Linder et al., 2006). A systematic review on the use of artificial neural networks in decision support in cancer by Lisboa et al. showed that the number of clinical trials (CTs) and randomised controlled trials (RCTs) involving the use of ANNs in diagnosis and prognosis has increased from 1 to 38 over the past decade. However, out of 396 studies involving the use of ANNs in cancer, only 27 were either CTs or RCTs. Out of these trials, 21 showed an increase in benefit to healthcare provision and 6 did not. None of these studies however showed a decrease in benefit. Overall, the reviewed publications support the neural network approach but while on the one hand they identify trends in areas of clinical promise (particularly diagnosis, prognosis and therapeutic guidance for cancer), on the other they highlight the need for more extensive application of rigorous methodologies (Lisboa & Taktak, 2006). Interesting, a review on the use of ANNs in the field of Gastroenterology over the last 10 years (their application in the field of gastroenterology has now entered the second decade) underlines that the increasing complexity of clinical data requires the use of mathematical models that are able to capture the key properties of entire ensembles, including their linkages and their hubs, abandoning the traditional statistical reductionistic approach, which tends to ‘see’ things individually, to simplify and to look at one single element at a time (Pace & Savarino, 2007). Some authors, for example, assessed the performance of ANNs in recognizing patients with chronic atrophic gastritis, a state of chronic inflammation that can eventually progress to gastric carcinoma, by using only clinical and biochemical variables (Annibale & Lahner, 2007). In the field of urology, several papers have addressed the predictive efficacy of ANNs. In urological cancer, ANNs appear to be accurate and more explorative than traditional regression statistics artificial intelligence methods when used to analyze large data cohorts. Furthermore, they allow individualized prediction of disease behaviour. Each artificial intelligence method has characteristics that make it suitable for different tasks. The lack of transparency of ANNs hinders global scientific community acceptance, but this can be overcome by neuro-fuzzy modeling systems (Abbod et al., 2007). New biomarkers within multivariate models have been analyzed with ANNs to improve early detection of prostate cancer (Stephan et al., 2007). Another field of application is the management of urolithiasis, a worldwide clinical challenge embracing a multitude of difficulties in diagnosis, treatment and prevention of recurrence. Recent reports have examined the role of ANNs in prediction of stone presence and composition, spontaneous passage, clearance and re-growth after treatment (Rajan & Tolley, 2005). The results suggest that ANNs may prove useful in clinician-led decision-making processes. ANNs can identify important predictive variables and accurately predict treatment outcomes in clinical medicine but although the initial results appear promising, further prospective studies of larger patient cohorts will need to be performed in order to determine whether this mode of analysis can surpass standard statistical predictive methods, not only when solving problems concerning diagnosis and its classification into subtypes but also when predicting clinical outcomes of patients affected by diverse pathologies.

The Usefulness of Artificial Neural Networks in Predicting the Outcome of Hematopoietic Stem Cell Transplantation

219

1.2 Artificial neural networks in hematology and bone marrow transplantation Very few reports on this topic have been published in the field of hematology. The first computing devices based on artificial intelligence (AI) have been applied to routine laboratory data management whereas new innovative tools, based on neural networks trained with data from peripheral blood analysis, have been used for differential diagnosis in diseases such as anemias, thalassemias and leukemias. The introduction of the first microarray based and bio-informatic approach for molecular diagnosis of hematological malignancies can be considered a major step ahead. This approach is based on the monitoring of simultaneous expression of thousands of genes using DNA microarray, independently of previous biological knowledge, analyzed using AI devices (Zini, 2005). In an attempt to create an application for discriminating different types of anemia, simply using data from peripheral blood, Zini & d’Onofrio (2003) collected data from peripheral blood of 1000 patients diagnosed mainly with hematopoietic disorders in 22 Italian Hematology Centers. The ANNs were trained with labeled samples and showed high capability of clustering signals according to the predefined normal as well as pathological profiles. In 2002, Amendolia et al. investigated the use of ANNs for the classification of thalassemic pathologies, exclusively using the hematologic parameters resulting from hemochromocytometric analysis. Different combinations of ANNs made it possible to discriminate thalassemia carriers from normals with 94% classification accuracy, 92% sensitivity, and 95% specificity. Based on these results, an automated system for real-time support in diagnoses was proposed (Amendolia et al., 2002). All these intriguing reports of ANNs in the field of hematology kindled our curiosity to discover whether ANNs were capable of predicting the outcome of hematopoietic stem cell transplantation (HSCT) after analyzing donor and recipient pre-transplantation clinical and immunogenetic variables. 1.3 The difficult setting of unrelated bone marrow transplantation in thalassemia. Patients with chronic non-malignant genetic disorders, such as thalassemia, are faced with a dramatic decision: they can either undergo HSCT with a good possibility of cure but a high chance of death or continue the more conventional treatment with blood transfusions and iron chelation therapy. The important advances made in conventional treatment now allow transfusion-dependent thalassemia patients to live much longer (Caocci et al., 2006; Borgna Pignatti et al., 2004) but as a result these patients must cope with complications that occur over time. Treatment may be required for heart or liver diseases, infections, osteoporosis and other serious health problems. On the other hand, although HSCT from an HLA-identical sibling can offer thalassemia patients a probability of cure that is close to 90% in children and adults in good clinical conditions (Lucarelli et al., 1990), this procedure is associated with a significant risk of mortality (Lucarelli et al., 1997), especially in patients with advanced age or poor clinical conditions. Moreover, the chance that any given sibling will be HLA matched with a potential recipient is one out of four which means that most patients will need to search for a compatible donor in the registries of voluntary donors worldwide. Transplantation from unrelated donors (UD) is burdened by an increased risk of acute and chronic graft-versus host disease (GVHD) with a consequent negative impact on overall survival (La Nasa et al., 2006). Therefore, every effort should be made to carefully evaluate the risk of GVHD before performing UD-HSCT (Hansen et al., 1998).

220

Artificial Neural Networks - Methodological Advances and Biomedical Applications

Graft-versus-host disease remains the major barrier to the successful outcome of HSCT in thalassemia. HLA disparity between the donor and the recipient is clearly the most powerful risk factor but also older age, gender mismatch, Pesaro risk class, cytomegalovirus (CMV) positivity as well as higher median infused hematopoietic stem cell doses have been shown to increase the risk for GVHD (Lucarelli et al., 1996). Evidence emerging from recent reports indicates a correlation between certain immunogenetic variables and the occurrence of GVHD: donor-recipient HLA-Cw ligand groups for killer immunoglobulin-like receptors (KIRs), KIR genotypes, the HLA-G 14-basepair (bp) polymorphism and HLA-DPB1 disparity (La Nasa et al., 2007; Littera et al., 2010; Fleischhauer et al., 2006). Although this information may contribute to our understanding of the pathogenesis of GVHD, it is difficult to apply in clinical practice. What we need is a simple prognostic tool capable of analyzing the most relevant predictive variables.

Fig. 1. To gaze into a crystal ball for a glimpse of the future has always been the dream of every doctor. Reliable assessment of the acute GVHD risk is crucial for making rational treatment decisions. During the process of donor selection and before discussing the choice of treatment with patients and their relatives, it is essential for physicians to integrate their knowledge with statistical or algorithmic tools capable of accurately predicting the likely incidence of GVHD. A more accurate prediction of acute and chronic GVHD would not only improve GVHD prophylaxis and conditioning regimens, but would also allow physicians to adapt their communication practices appropriately and to ensure that patients are supplied with effective

The Usefulness of Artificial Neural Networks in Predicting the Outcome of Hematopoietic Stem Cell Transplantation

221

and comprehensive information on the pros and cons of transplantation, including the possibility of dying. This is particularly relevant for patients with chronic non malignant disorders, such as thalassemia. Maybe ANNs represent the crystal ball we are all looking for.

2. Patients and methods
2.1 Patients We compared the prognostic performance of ANNs versus LR for predicting acute GVHD in a group of 78 beta-thalasssemia major patients given UD-HSCT (Caocci et al, 2010). The following clinical and immunogenetic paramenters were considered: recipient gender, recipient age, donor gender, donor age, the combination male recipient/female donor versus the other possible combinations, recipients and/or donors with positive CMV serology versus donor and recipient pairs with negative CMV serology, the Pesaro risk class at transplantation, HCV-RNA positivity, median infused CD34 cell dose, Treosulphan-containing conditioning regimen versus other regimens, HLA Class I mismatch, presence of HLA-A11, non permissive HLA-DPB1 disparity in GVHD direction, presence of the HLA-G 14-basepair deletion/deletion polymorphism in recipients, presence of the HLA-G 14-basepair deletion/deletion polymorphism in donors, heterozygosity for HLA-Cw ligand groups 1 and 2 in patients, recipient KIR ligand/donor activating KIR (recipient C1 absent/donor KIR2DS2 present versus the other 3 combinations; recipient C2 absent/donor KIR2DS1 present versus the other 3 combinations), recipient KIR ligand/donor inibitory KIR (patient C1 absent/donor KIR2DL2 present versus the other 3 combinations; recipient C1 absent/donor KIR2DL3 present versus the other 3 combinations; patient C2 absent/donor KIR2DL1 present versus the other 3 combinations), donor homozygosity for KIR haplotype A (Uhrberg, 2002; Colonna, 1995; Bassi, 2007; Cook, 2004; Harrison, 1993). 2.2 Statistical analysis Patient, disease, and transplantation-related variables were expressed as median and range or percentage, as appropriate. For the HSCT outcome, patients were censored at the time of rejection, death, or last follow-up. Probabilities of overall survival (OS) and survival with transfusion independence (thalassemia-free survival) were estimated by the Kaplan-Meier method. The following 24 independent variables were analyzed for their potential impact on aGVHD: recipient gender, recipient age, donor gender, donor age, the combination male recipient/female donor versus the other possible combinations, recipients and/or donors with positive CMV serology versus donor and recipient pairs with negative CMV serology, the Pesaro risk class at HSCT, HCV-RNA positivity, median infused CD34 cell dose, Treosulphan conditioning regimen versus other regimens, HLA Class I mismatch, presence of HLA-A11, non permissive HLA-DPB1 disparity in GVHD direction, presence of the HLAG 14-basepair deletion/deletion polymorphism in recipients, presence of the HLA-G 14basepair deletion/deletion polymorphism in donors, heterozygosity for HLA-Cw ligand groups 1 and 2 in patients, recipient KIR ligand/donor activatory KIR (recipient C1 absent/donor KIR2DS2 present versus the other 3 combinations; recipient C2 absent/donor KIR2DS1 present versus the other 3 combinations), recipient KIR ligand/donor inibitory KIR (patient C1 absent/donor KIR2DL2 present versus the other 3 combinations; recipient

222

Artificial Neural Networks - Methodological Advances and Biomedical Applications

C1 absent/donor KIR2DL3 present versus the other 3 combinations; patient C2 absent/donor KIR2DL1 present versus the other 3 combinations), donor homozygosity for KIR haplotype A. 2.3 Logistic regression A binomial LR model with 24 independent variables (3 continuous and 21 categorical) was developed (Table 2). Acute GVHD was considered as a dichotomous dependent variable. Five consecutive random extractions were performed. For each extraction, patients were split into a learning data set (68 patients) and a test data set (10 patients). The independent variables were fitted into LR models via forward likelihood ratio test (chi-square difference) and stepwise selection. The chi-square test proposed by Hosmer and Lemeshow was used to analyze the goodness of fit: a finding of non-significance corresponds to the conclusion that the model adequately fits the data. Variables were retained only if their resulting p-value was 0.05. The final equation, developed through parameter estimates with standard errors, odds ratios and asymptotic 95% confidence intervals for all significant variables calculated, was applied to each case of the data test. A cut-off value of 0.5 was established for assigning the probability of GVHD: “GVHD yes” (1) or “GVHD no” (0). Sensitivity and specificity were determined in the learning and test data sets of each random extraction, sensitivity being the ratio between true positive and true negative plus false negative and specificity the ratio between true negative and true negative plus false positive. Mean sensitivity and specificity of LR obtained in five consecutive extractions were compared to ANN, using the chi-square test with Yate’s correction. Statistical analysis was performed using SPSS® software, version 12 (SPSS Inc., Chicago, IL, USA) 2.4 Artificial neural networks ANNs are capable of learning from observed data or examples and under certain conditions are able to approximate nonlinear functions with arbitrary precision. The technique was originally inspired by perceptions of how the human brain learns and processes information and since then has successfully been applied in many different fields, including mathematics, engineering, medicine, economics, meteorology, psychology, neurology, and many others. Although the predictive power of ANNs is often superior to that of other more traditional methods, they are still regarded as black-boxes where it is difficult for the user to gain insight into the influence of the independent variables in the prediction process. While ANNs are capable of learning the relationship between the input parameters and the controlled and uncontrolled variables, they do not generate information on the causal relationship between the input and output patterns. Several studies are currently underway to overcome this problem. The structure of ANN usually consists of three layers (Fig. 2). The input layer accepts data sets from an external source that constitute inputs to the next layer of neurons. The next layer is called the hidden layer because its neuron values are not visible outside the net. The use of one or more hidden layers increases the net’s learning abilities. The final layer is the output layer. Each single neuron is connected to the neurons of the previous layer through adaptable synaptic weights. Knowledge is generally stored as distributed patterns of activation in weight matrices. The key feature of neural networks is that they learn the input/output relationship through training. The training data set includes a number of cases, each containing values for a range

The Usefulness of Artificial Neural Networks in Predicting the Outcome of Hematopoietic Stem Cell Transplantation

223

Fig. 2. The three layers structure of ANN of well-matched input and output variables. The weighted connections between neurons in each layer are adjusted by a training algorithm to minimize error and provide accurate predictions on the training set. The outputs are the dependent variables that the network produces for the corresponding input. It is important that all learning data is supplied to the network as a data set. Once the input is propagated to the output neuron, this neuron compares its activation with the expected training output. If there is an error, the output neuron adjusts the connection weights to compensate the error by going backwards through the network. This step by step process is called backpropagation. The backpropagation (BP) algorithm and its variants are the most powerful learning algorithms in neural networks. By calculating the gradient vector of the error surface, the error gradually declines until all the expected outputs are correctly displayed. The Neural Network ToolboxTM 6 of the software Matlab® 2008, version 7.6 (MathWorks, inc.) was used to develop a three layer feed forward neural network with the default tansigmoid transfer function in the hidden layer and linear transfer function in the output layer (Schwarzer et al., Demuth, 2008). The input layer of 24 neurons receives data that are processed in the hidden layer (30 neurons) and output layer (1 neuron). The output neuron predicts a number between 1 and 0 (goal), representing the event “GVHD yes” (1) or “GVHD no” (0), respectively. A cut-off value of 0.5 was established for assigning probability 1 or 0. The architecture of ANN is schematized in Fig. 3. Input neurons receive data represented by the values of 24 independent variables processed in the hidden layer. The meaning of this process is to calculate interconnection weights between variables with the purpose of predicting an outcome and to calculate an error value by comparing this output value with the known outcome. The ANN attempts to minimize the error by adjusting the weights according to a learning algorithm (Linder et al., 2006). For the training procedure, we applied the ‘on-line back-propagation’ method on the same 5 sets of 68 patients previously analyzed by LR.

224

Artificial Neural Networks - Methodological Advances and Biomedical Applications

The 5 test phases utilized 10 patients randomly extracted from the entire cohort and not used in the training phase. A standard error of less than 10-2 was required. Mean sensitivity and specificity of the 5 consecutive data sets were determined in the data test and compared to LR. Because sensitivity and specificity in the 5 learning tests always resulted to be 100%, they were considered not comparable to LR.

Fig. 3. Architecture of the three-layer artificial neural network. The input layer of 24 neurons (independent variables) receives data that are processed in the hidden layer and output layer (1 neuron). The output neuron predicts a number between 1 and 0 (goal), representing the event “GVHD yes” (1) or “GVHD no” (0), respectively.

3. Results
Three-year Kaplan-Meier estimates for the 78 patients studied were 89.7% for survival, 76.9% for thalassemia-free survival, 11.5% for the cumulative incidence of rejection and 10.3% for TRM. Nine patients rejected the allograft and 7 died of transplantation-related complications (Figure 4). Twenty-six patients (33.3%) developed grade II-IV acute GVHD (Figure 5). In multivariate analysis, only donor KIR AA haplotypes were independently significantly correlated to acute GVHD in our cohort of 78 patients (p=0.037). However, we found a

The Usefulness of Artificial Neural Networks in Predicting the Outcome of Hematopoietic Stem Cell Transplantation

225

positive trend for donor age (p=0.51), patient heterozygosity (C1/C2) for the HLA-Cw KIR ligands (p=0.56) and donor homozygosity (deletion/deletion) for the HLA-G 14-bp polymorphism (p=0.57) (Table 2).

Fig. 4. Kaplan-Meier probabilities of overall survival, thalassemia-free survival, cumulative incidence of mortality and rejection in 78 thalassemia patients transplanted from an unrelated donor. Table 3 shows the prognostic performance of LR and ANN in predicting acute GVHD in 5 consecutive randomly extracted training and test data sets. Sensitivity and specificity were determined in the learning and test data sets of each random extraction. Comparisons between LR and ANN on training data sets (5 consecutive extractions each composed of 68 patients) were not considered since ANN was able to recognize 100% of correct events by

226

Artificial Neural Networks - Methodological Advances and Biomedical Applications

means of its peculiar learning algorithm, whereas LR showed a mean value of 88.5% for specificity and 36.4% for sensitivity.

Fig. 5. Kaplan-Meier probabilities of cumulative incidence of acute GVHD in 78 thalassemia patients transplanted from an unrelated donor In test data sets (5 extractions each composed of 10 patients), the mean specificity of LR was 80.5% compared to 90.1% of ANN (capability of predicting the absence of acute GVHD in patients who did not experience acute GVHD); this difference was not statistically significant. The mean sensitivity of LR was 21.7% compared to 83.3% of ANN (capability of predicting acute GVHD in patients who developed acute GVHD after HSCT). This difference was statistically significant (p

Artificial Neural Network for Biomedical Purpose

Similar Documents

Diabetic Retinopathy Literature Review

Artificial Neural Network Essentials

Unknown

Models for Estimation of Isometric Wrist Joint Torques Using Surface Electromyography

Hybrid Machining Research Paper

Data Warehousing and Data Mining

Demo

Philosophy

Nit-Silchar B.Tech Syllabus

Data Mining in Hospitals

A Fusion Approach for EfﬁCient Human Skin Detection

Biometrics

New Medical Devices in the Us

Idrivesa

An Argumentative Research Paper on Research Topic: What Is Science?

Popular Essays