Neural Network Doc 1y1m4d

ABSTRACT A neural network is a data processing system consisting of a large number of simple, highly interconnected processing elements in an architecture inspired by the structure of the cerebral cortex portion of the brain. Hence, neural networks are often capable of doing things which humans or animals do well but which conventional computers often do poorly. Neural networks have emerged in the past few years as an area of unusual opportunity for research, development and application to a variety of real world problems. Indeed, neural networks exhibit characteristics and capabilities not provided by any other technology. Examples include reading Japanese Kanji characters and human handwriting, reading typewritten text, compensating for alignment errors in robots, interpreting very “noisy” signals (e.g. electrocardiograms), modeling complex systems that cannot be modelled mathematically, and predicting whether proposed loans will be good or fail.This paper presents a brief tutorial on neural networks and briefly describes several applications

ARTIFICIAL NEURAL-NETWORKS

CHAPTER 1 INTRODUCTION TO NEURAL NETWORKS 1.1 What is a Neural Network? An Artificial Neural Network (ANN) is an information processing paradigm that is inspired by the way biological nervous systems, such as the brain, process information. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems. ANNs, like people, learn by example. An ANN is configured for a specific application, such as pattern recognition or data classification, through a learning process. Learning in biological systems involves adjustments to the synaptic connections that exist between the neurones. This is true for ANNs as well.

1.2 Historical background Neural network simulations appear to be a recent development. However, this field was established before the advent of computers, and has survived at least one major setback and several eras. Many important advances have been boosted by the use of inexpensive computer emulations. Following an initial period of enthusiasm, the field survived a period of frustration and disrepute. During this period when funding and professional was minimal, important advances were made by relatively few researchers. These pioneers were able to develop convincing technology which sured the limitations identified by Minsky and Papert. Minsky and Papert, published a book (in 1969) in which they summed up a general feeling of frustration (against neural networks) among researchers, and was thus accepted by most without further analysis. Currently, the neural network field enjoys a resurgence of interest and a corresponding increase in funding. The first artificial neuron was produced in 1943 by the neurophysiologist Warren McCulloch and the logician Walter Pits. But the technology available at that time did not allow them to do too much.

Department of ECE,MIST Page 2


1.3 Why use neural networks? Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A trained neural network can be thought of as an "expert" in the category of information it has been given to analyze. This expert can then be used to provide projections given new situations of interest and answer "what if" questions. Other advantages include: 1. Adaptive learning: An ability to learn how to do tasks based on the data given for training or 2. initial experience. 3. Self-Organization: An ANN can create its own organization or representation of the information it receives during learning time. 4. Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability 5. Fault Tolerance via Redundant Information Coding: Partial destruction of a network leads to the corresponding degradation of performance. However, some network capabilities may be retained even with major network damage.

1.4 Neural networks versus conventional computers Neural networks take a different approach to problem solving than that of conventional computers. Conventional computers use an algorithmic approach i.e. the computer follows a set of instructions in order to solve a problem. Unless the specific steps that the computer needs to follow are known the computer cannot solve the problem. That restricts the problem solving capability of conventional computers to problems that we already understand and know how to solve. But computers would be so much more useful if they could do things that we don't exactly know how to do. Neural networks process information in a similar way the human brain does. The network is composed of a large number of highly interconnected processing elements (neurones) working in parallel to solve a specific problem. Neural networks learn by example. They cannot be programmed to perform a specific task. The examples must be selected carefully otherwise useful time is wasted or even worse the network might


ARTIFICIAL NEURAL-NETWORKS be functioning incorrectly. The disadvantage is that because the network finds out how to solve the problem by itself, its operation can be unpredictable. On the other hand, conventional computers use a cognitive approach to problem solving; the way the problem is to solved must be known and stated in small unambiguous instructions. These instructions are then converted to a high level language program and then into machine code that the computer can understand. These machines are totally predictable; if anything goes wrong is due to a software or hardware fault. Neural networks and conventional algorithmic computers are not in competition but complement each other. There are tasks are more suited to an algorithmic approach like arithmetic operations and tasks that are more suited to neural networks. Even more, a large number of tasks, require systems that use a combination of the two approaches (normally a conventional computer is used to supervise the neural network) in order to perform at maximum efficiency.



CHAPTER 2 HUMAN AND ARTIFICIAL NEURONS INVESTIGATING THE SIMILARITIES 2.1 How the Human Brain Learns? Much is still unknown about how the brain trains itself to process information, so theories abound. In the human brain, a typical neuron collects signals from others through a host of fine structures called dendrites. The neuron sends out spikes of electrical activity through a long, thin stand known as an axon, which splits into thousands of branches. At the end of each branch, a structure called a synapse converts the activity from the axon into electrical effects that inhibit or excite activity from the axon into electrical effects that inhibit or excite activity in the connected neurones. When a neuron receives excitatory input that is sufficiently large compared with its inhibitory input, it sends a spike of electrical activity down its axon. Learning occurs by changing the effectiveness of the synapses so that the influence of one neuron on another changes.

Fig 1: Components of neuron

2.2 From Human Neurons to Artificial Neurons We conduct these neural networks by first trying to deduce the essential features of neurones and their interconnections. We then typically program a computer to simulate these features. However because our knowledge of neurones is incomplete and our computing power is limited, our models are necessarily gross idealizations of real networks of neurones.



CHAPTER 3 AN ENGINEERING APPROACH 3.1 A Neuron A more sophisticated neuron is the McCulloch and Pitts model (M). The difference from the previous model is that the inputs are ‘weighted’; the effect that each input has at decision making is dependent on the weight of the particular input. The weight of an input is a number which when multiplied with the input gives the weighted input. These weighted inputs are then added together and if they exceed a pre-set threshold value, the neuron fires. In any other case the neuron does not fire. In mathematical , the neuron fires if and only if; the addition of input weights and of the threshold makes this neuron a very flexible and powerful one. The M neuron has the ability to adapt to a particular situation by changing its weights and/or threshold. Various algorithms exist that cause the neuron to 'adapt'; the most used ones are the Delta rule and the back error propagation. The former is used in feed-forward networks and the latter in networks.

y

y u

x1

u w0 wi xi

xn Fig 2: Neuron Model



CHAPTER 4 ARCHITECTURE OF NEURAL NETWORKS 4.1 Feed-forward networks Feed-forward ANNs allow signals to travel one way only; from input to output. There is no (loops) i.e. the output of any layer does not affect that same layer. Feed-forward ANNs tend to be straight forward networks that associate inputs with outputs. They are extensively used in pattern recognition. This type of organization is also referred to as bottom-up or top-down.

Fig 3: Artificial neuron model

4.2 networks networks (figure 1) can have signals traveling in both directions by introducing loops in the network. networks are very powerful and can get extremely complicated. networks are dynamic; their 'state' is changing continuously until they reach an equilibrium point. They remain at the equilibrium point until the input changes and a new equilibrium needs to be found. architectures are also referred to as interactive or recurrent, although the latter term is often used to denote connections in single-layer organizations.



Fig 4: Networks

4.3 Network layers The commonest type of artificial neural network consists of three groups, or layers, of units: a layer of "input" units is connected to a layer of "hidden" units, which is connected to a layer of "output" units. The activity of the input units represents the raw information that is fed into the network. The activity of each hidden unit is determined by the activities of the input units and the weights on the connections between the input and the hidden units. The behavior of the output units depends on the activity of the hidden units and the weights between the hidden and output units. This simple type of network is interesting because the hidden units are free to construct their own representations of the input. The weights between the input and hidden units determine when each hidden unit is active, and so by modifying these weights, a hidden unit can choose what it represents. We also distinguish single-layer and multi-layer architectures. The single-layer organization, in which all units are connected to one another, constitutes the most general case and is of more potential computational power than hierarchically structured multi-layer organizations. In multi-layer networks, units are often numbered by layer, instead of following a global numbering.



4.4 Perceptrons The most influential work on neural nets in the 60's went under the heading of 'perceptrons' a term coined by Frank Rosenblatt. The perceptron

turns out to be an

M model (neuron with weighted inputs) with some additional, fixed, pre-processing. Units labeled A1, A2, Aj, Ap are called association units and their task is to extract specific, localized featured from the input images. Perceptrons mimic the basic idea behind the mammalian visual system. They were mainly used in pattern recognition even though their capabilities extended a lot more. In 1969 Minsky and Papert wrote a book in which they described the limitations of single layer Perceptrons. The impact that the book had was tremendous and caused a lot of neural network researchers to loose their interest. The book was very well written and showed mathematically that single layer perceptrons could not do some basic pattern recognition operations like determining the parity of a shape or determining whether a shape is connected or not. What they did not realize, until the 80's, is that given the appropriate training, multilevel perceptrons can do these operations.

Fig 5: Perceptron Process



CHAPTER 5 THE LEARNING PROCESS 5.1 Types of learning All learning methods used for adaptive neural networks can be classified into two major categories: Supervised learning which incorporates an external teacher, so that each output unit is told what its desired response to input signals ought to be. During the learning process global information may be required. Paradigms of supervised learning . An important issue concerning supervised learning is the problem of error convergence, i.e. the minimization of error between the desired and computed unit values. The aim is to determine a set of weights which minimizes the error. One well-known method, which is common to many learning paradigms, is the least mean square (LMS) convergence. Unsupervised learning uses no external teacher and is based upon only local information. It is also referred to as self-organization, in the sense that it selforganizes data presented to the network and detects their emergent collective properties. Paradigms of unsupervised learning are Hebbian learning . Human Neurons to Artificial Neuron Esther aspect of learning concerns the distinction or not of a separate phase, during which the network is trained, and a subsequent operation phase. We say that a neural network learns off-line if the learning phase and the operation phase are distinct. A neural network learns on-line if it learns and operates at the same time. Usually, supervised learning is performed offline, whereas unsupervised learning is performed on-line.



dw E w  dt

E

Fig 6: Unsupervised learning graph

5.2 Transfer Function: The behavior of an ANN (Artificial Neural Network) depends on both the weights and the input-output function (transfer function) that is specified for the units. This function typically falls into one of three categories:  linear (or ramp)  threshold  sigmoid

f  a   1  1  exp(  a ) 

For linear units, the output activity is proportional to the total weighted output. For threshold units, the output are set at one of two levels, depending on whether the total input is greater than or less than some threshold value. For sigmoid units, the output varies continuously but not linearly as the input changes. Sigmoid units bear a greater resemblance to real neurones than do linear or threshold units, but all three must be considered rough approximations. We can teach a three-layer network to perform a particular task by using the following procedure: We present the network with training examples, which consist of a pattern of activities for the input units together with the desired pattern of activities for the output units. We determine how closely the actual output of the network matches the


ARTIFICIAL NEURAL-NETWORKS desired output. We change the weight of each connection so that the network produces a better approximation of the desired output. In order to train a neural network to perform some task, we must adjust the weights of each unit in such a way that the error between the desired output and the actual output is reduced. This process requires that the neural network compute the error derivative of the weights (EW). In other words, it must calculate how the error changes as each weight is increased or decreased slightly. The back propagation algorithm is the most widely used method for determining the EW. The back-propagation algorithm is easiest to understand if all the units in the network are linear. The algorithm computes each EW by first computing the EA, the rate at which the error changes as the activity level of a unit is changed. For output units, the EA is simply the difference between the actual and the desired output. To compute the EA for a hidden unit in the layer just before the output layer, we first identify all the weights between that hidden unit and the output units to which it is connected. We then multiply those weights by the EAs of those output units and add the products. This sum equals the EA for the chosen hidden unit. After calculating all the EAs in the hidden layer just before the output layer, we can compute in like fashion the EAs for other layers, moving from layer to layer in a direction opposite to the way activities propagate through the network. This is what gives back propagation its name. Once the EA has been computed for a unit, it is straight forward to compute the EW for each incoming connection of the unit. The EW is the product of the EA and the activity through the incoming connection.


ARTIFICIAL NEURAL-NETWORKS Fig 7: Back Propagation Graph

CHAPTER 6 DATA FILTERS That is all we need to know about Neural Nets, if we want to make forecasts. The next problem with which you will have to fight - is noise in real signals. And the biggest trouble, that nobody knows what is noise and what is pure signal. For example, how to choose the right variant for the problem in the right

Fig 8: Random Signal

6.1 Adaptive filters Main features, which we must obtain: 1) Zero-phase distortion (because we are going to make forecasts and we shouldn’t influence on the data’s phase we have. 2) We don’t know what real signal is and what is noise, especially in case of share values or smth like that. So one have to decide apriori what is the signal we want to obtain. In this chapter I’m going to make an overview of 3 types of adaptive filters: Zerophase filter (smoothing filter), Kalman filter (error estimator), and Empirical mode decomposition.



6.2 Zero-phase filter The main equation of this filter: y ( n)  b(1) x ( n)  b( 2) x ( n  1)  ...  b( nb  1) x ( n  nb)  a ( 2) y ( n  1)  ...  a ( na  1) y ( n  na )

The scheme of this filter b(1 )



X(n) -1

Z

X(n-1)

Z-1

X(n-2)

y(n)

b(2 )

a(2)

b( 3)

a(3)

-1

Z

Z-1

…

y(n-1) y(n-2) …

X(n-nb)

-1

Z

a(na+1 )

b(nb+1 )

-1

Z

Fig 9: Zero Phase Filter Where x(n) is the signal we have at the moment n . Z-1 – means that we take the previous value of the signal (moment (n-1)). b ,

a - are the filter coefficients. Filter

performs zero-phase digital filtering by processing the input data in both the forward and reverse directions. After filtering in the forward direction, it reverses the filtered sequence and runs it back through the filter. The resulting sequence has precisely zero-phase distortion and double the filter order. filter minimizes start-up and ending transients by matching initial conditions, and works for both real and complex inputs. Note that filter should not be used with differentiator and Hilbert FIR filters, since the operation of these filters depends heavily on their phase response. Problem: After the neural net the forecast will have 1 day delay if we tried to make forecast for one day. It looks like the picture on the right. But I should mention that all statistical values e.g. correlation coefficient are very good. It means that it’s is possible to use this filter, but with some kind of error estimator.


y(n-na)


6.3 Kalman filter (error estimator) Consider a linear, discrete-time dynamical system. The concept of state is fundamental to this description. The state vector or simply state, denoted by x, is defined as the minimal set of data that is sufficient to uniquely describe the unforced dynamical behavior of the system; the subscript n denotes discrete time. In other words, the state is the least amount of data on the past behavior of the system that is needed to predict its future behavior. Typically, the state xk is unknown. To estimate it, we use a set of observed data, denoted by the vector y k. In mathematical , the block diagram embodies the following pair of equations:

Fig 10: Pattern Graph

y (n)

+

K( n)



x(n  1)

+

Z-1 

ac x (n  1)

c

a



x( n  1)

Fig 11: Kalman Filter



The equation of this filter is given below:

   x ( n)  a x ( n  1)  k ( n)[ y ( n)  ac x ( n  1)], where x ( n)  ax ( n  1)  w( n  1) model of generating w( n )  white noise and y ( n)  cx ( n)   ( n ) signal after neural net , ( n)  white noise

Problem: After Kalman filter the error became less, but there is still a day delay. The solution is Empirical Mode Decomposition.

6.4 Empirical Mode Decomposition A new nonlinear technique, referred to as Empirical Mode Decomposition (EMD), has recently been pioneered by N.E. Huang et al. for adaptively representing nonstationary signals as sums of zero-mean AMFM components [2]. Although it often proved remarkably effective [1, 2, 5, 6, 8], the technique is faced with the difficulty of being essentially defined by an algorithm, and therefore of not itting an analytical formulation which would allow for a theoretical analysis and performance evaluation. The purpose of this paper is therefore to contribute experimentally to a better understanding of the method and to propose various improvements upon the original formulation. Some preliminary elements of experimental performance evaluation will also be provided for giving a flavor of the efficiency of the decomposition, as well as of the difficulty of its interpretation. The starting point of the Empirical Mode Decomposition (EMD) is to consider oscillations in signals at a very local level. In fact, if we look at the evolution of a signal x(t) between two consecutive extrema (say, two minima occurring at times t− and t+), we can heuristically define a (local) high-frequency part {d(t), t− ≤ t ≤ t+}, or local detail, which corresponds to the oscillation terminating at the two minima and ing through the maximum which necessarily exists in between them. For the picture to be complete, one still has to identify the corresponding (local) lowfrequency part m(t), or local trend, so that we have x(t) = m(t) + d(t) for t− ≤ t ≤ t+. Assuming that this is done in some proper way for all the oscillations composing the entire signal, the procedure can then be applied on the residual consisting of


ARTIFICIAL NEURAL-NETWORKS all local trends, and constitutive components of a signal can therefore be iteratively extracted. Given a signal x(t), the effective algorithm of EMD can be summarized as follows:

1. Identity all extrema of x(t) 2. Interpolate between minima (resp. maxima), ending up with some envelope emin(t) (resp. emax(t)) 3. Compute the mean m(t) = (emin(t)+emax(t))/2 4. Extract the detail d(t) = x(t) − m(t) 5. Iterate on the residual m(t) In practice, the above procedure has to be refined by a sifting process which amounts to first iterating steps 1 to 4 upon the detail signal d(t), until this latter can be considered as zero-mean according to some stopping criterion. Once this is achieved, the detail is referred to as an Intrinsic Mode Function (IMF), the corresponding residual is computed and step 5 applies. By construction, the number of extrema is decreased when going from one residual to the next, and the whole decomposition is guaranteed to be completed with a finite number of modes. Modes and residuals have been heuristically introduced on “spectral” arguments, but this must not be considered from a too narrow perspective. First, it is worth stressing the fact that, even in the case of harmonic oscillations, the high vs. low frequency discrimination mentioned above applies only locally and corresponds by no way to a pre-determined sub band filtering (as, e.g., in a wavelet transform). Selection of modes rather corresponds to an automatic and adaptive (signal-dependent) time-variant filtering. Let’s try to do this procedure with next signal x  0,7 sin( 20t )  0,5 sin( 400t ), t  [01]



Fig 12:Iteration for signal x  0,7 sin( 20t )  0,5 sin(400t ), t  [01]

Fig 13: Iteration and residue

Fig 14: filter from high frequency part (light gray – EMD, dark gray – zero phase filters) Problem: Using this method we have strong boundary effects. Sifting process can minimize this effect, but not at all.

6.5 Zero-phase filter +Kalman filter +EMD = Solution The so called solution is next. We should use data, obtained from zero-phase filter on the boundaries, we should use linear weighed sum of zero-phase and some IMF’s in the middle. And after the forecast is done, we should apply Kalman filter to the


ARTIFICIAL NEURAL-NETWORKS forecast. Let’s try to do that. The figure on the bottom is the forecast of some company’s share value (production set). There is no delay.

Fig 15: Variables through Pattern

6.6Holder-function behavior is important The main idea is next. Consider f (t )  D f Holder derived, that | f (t  t )  f (t ) | const (t ) ( t ) ,  (t )  [0,1]   0 means that we have break of second order   1means that we have O (t )

So this formula is a somewhat connection between “bad” functions and “good” functions. If we will look on this formula with more precise we will notice, that we can catch moments in time, when our function knows, that it’s going to change it’s behavior from one to another. It means that today we can make a forecast on tomorrow behavior. But one should mention that we don’t know the sigh on what behavior is



CHAPTER 7 APPLICATIONS The utility of artificial neural network models lies in the fact that they can be used to infer a function from observations. This is particularly useful in applications where the complexity of the data or task makes the design of such a function by hand impractical.

Real-life applications The tasks artificial neural networks are applied to tend to fall within the following broad categories: 

Function approximation, or regression analysis, including time series prediction, fitness approximation and modeling



Classification,

including pattern and

sequence

recognition, novelty

detection and sequential decision making 

Data processing, including filtering, clustering, blind source separation and compression



Robotics, including directing manipulators, prosthesis.



Control, including Computer numerical control Application areas include the system identification and control (vehicle control,

trajectory prediction,

process control, natural resources management), quantum

chemistry, game-playing and decision making (backgammon, chess, poker), pattern recognition (radar systems, face identification, object recognition and more), sequence recognition (gesture, speech, handwritten text recognition), medical diagnosis, financial applications (e.g. automated trading systems), data mining (or knowledge discovery in databases, "KDD"), visualization and e-mail spam filtering. Artificial neural networks have also been used to diagnose several cancers. An ANN based hybrid lung cancer detection system named HLND improves the accuracy Department of ECE,MIST Page 20

ARTIFICIAL NEURAL-NETWORKS of diagnosis and the speed of lung cancer radiology. These networks have also been used to diagnose prostate cancer. The diagnoses can be used to make specific models taken from a large group of patients compared to information of one given patient. The models do not depend on assumptions about correlations of different variables. Colorectal cancer has also been predicted using the neural networks. Neural networks could predict the outcome for a patient with colorectal cancer with more accuracy than the current clinical methods. After training, the networks could predict multiple patient outcomes from unrelated institutions.

7.1 Neural networks and neuroscience Theoretical and computational neuroscience is the field concerned with the theoretical analysis and the computational modeling of biological neural systems. Since neural systems are intimately related to cognitive processes and behavior, the field is closely related to cognitive and behavioral modeling. The aim of the field is to create models of biological neural systems in order to understand how biological systems work. To gain this understanding, neuroscientists strive to make a link between observed biological processes (data), biologically plausible mechanisms for neural processing and learning (biological neural network models) and theory (statistical learning theory and information theory).

7.2 Types of models Many models are used in the field, defined at different levels of abstraction and modeling different aspects of neural systems. They range from models of the shortterm behavior of individual neurons (e.g.,models of how the dynamics of neural circuitry arise from interactions between individual neurons and finally to models of how behavior can arise from abstract neural modules that represent complete subsystems. These include models of the long-term, and short-term plasticity, of neural systems and their relations to learning and memory from the individual neuron to the system level.

7.3 Memory networks Integrating external memory components with artificial neural networks has a long history dating back to early research in distributed representations andselforganizing maps. E.g. in sparse distribute memory the patterns encoded by neural Department of ECE,MIST Page 21

ARTIFICIAL NEURAL-NETWORKS networks are used as memory addresses for content-addressable memory, with "neurons" essentially serving as address encoders and decoders.

More recently deep learning was shown to be useful in semantic hashing where a deep graphical modelof the word-count vectors is obtained from a large set of documents. Documents are mapped to memory addresses in such a way that semantically similar documents are located at nearby addresses. Documents similar to a query document can then be found by simply accessing all the addresses that differ by only a few bits from the address of the query document.Neural Turing Machines developed by Google DeepMind extend the capabilities of deep neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples. Memory Networks is another extension to neural networks incorporating long-term memory which was developed by Facebook research. The long-term memory can be read and written to, with the goal of using it for prediction. These models have been applied in the context of question answering (QA) where the long-term memory effectively acts as a (dynamic) knowledge base, and the output is a textual response.

7.4 A Simple Network To Classify Hand Written Signals: Having defined neural networks, let's return to handwriting recognition. We can split the problem of recognizing handwritten digits into two sub-problems. First, we'd like a way of breaking an image containing many digits into a sequence of separate images, each containing a single digit. For example, we'd like to break the image



into six separate images,

We humans solve this segmentation problem with ease, but it's challenging for a computer program to correctly break up the image. Once the image has been segmented, the program then needs to classify each individual digit. So, for instance, we'd like our program to recognize that the first digit above,

is a 5.We'll focus on writing a program to solve the second problem, that is, classifying individual digits. We do this because it turns out that the segmentation problem is not so difficult to solve, once you have a good way of classifying individual digits. There are many approaches to solving the segmentation problem. One approach is to trial many different ways of segmenting the image, using the individual digit classifier to score each trial segmentation. A trial segmentation gets a high score if the individual digit classifier is confident of its classification in all segments, and a low score if the classifier is having a lot of trouble in one or more segments. The idea is that if the classifier is having trouble somewhere, then it's probably having trouble because the segmentation has been chosen incorrectly. This idea and other variations can be used to solve the segmentation problem quite well. So instead of worrying about segmentation we'll concentrate on developing a neural network which can solve the more interesting and difficult problem, namely, recognizing individual handwritten digits. To recognize individual digits we will use a three-layer neural network:



Fig 16: Recognizing Individual Digits Using a Three Layer Network The input layer of the network contains neurons encoding the values of the input pixels. As discussed in the next section, our training data for the network will consist of many 2828 by 2828 pixel images of scanned handwritten digits, and so the input layer contains 784=28×28784=28×28 neurons. For simplicity I've omitted most of the 784784 input neurons in the diagram above. The input pixels are greyscale, with a value of 0.00.0 representing white, a value of 1.01.0representing black, and in between values representing gradually darkening shades of grey. To understand why we do this, it helps to think about what the neural network is doing from first principles. Consider first the case where we use 1010 output neurons. Let's concentrate on the first output neuron, the one that's trying to decide whether or not the digit is a 00. It does this by weighing up evidence from the hidden layer of neurons. What are those hidden neurons doing? Well, just suppose for the sake of argument that the first neuron in the hidden layer detects whether or not an image like the following is present:



It can do this by heavily weighting input pixels which overlap with the image, and only lightly weighting the other inputs. In a similar way, let's suppose for the sake of argument that the second, third, and fourth neurons in the hidden layer detect whether or not the following images are present:

As you may have guessed, these four images together make up the 00image that we saw in the line of digits shown earlier:

So if all four of these hidden neurons are firing then we can conclude that the digit is a 00. Of course, that's not the only sort of evidence we can use to conclude that the image was a 00 - we could legitimately get a 00 in many other ways (say, through


ARTIFICIAL NEURAL-NETWORKS translations of the above images, or slight distortions). But it seems safe to say that at least in this case we'd conclude that the input was a 00. Supposing the neural network functions in this way, we can give a plausible explanation for why it's better to have 1010 outputs from the network, rather than 44. If we had 44 outputs, then the first output neuron would be trying to decide what the most significant bit of the digit was. And there's no easy way to relate that most significant bit to simple shapes like those shown above. It's hard to imagine that there's any good historical reason the component shapes of the digit will be closely related to (say) the most significant bit in the output. Now, with all that said, this is all just a heuristic. Nothing says that the three-layer neural network has to operate in the way I described, with the hidden neurons detecting simple component shapes. Maybe a clever learning algorithm will find some assignment of weights that lets us use only 44 output neurons. But as a heuristic the way of thinking I've described works pretty well, and can save you a lot of time in deg good neural network architectures. 

There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 33 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.990.99, and incorrect outputs have activation less than 0.010.01.

Fig 17:Digit Recognition Department of ECE,MIST Page 26

ARTIFICIAL NEURAL-NETWORKS Now that we have a design for our neural network, how can it learn to recognize digits? The first thing we'll need is a data set to learn from - a so-called training data set. We'll use the MNIST data set, which contains tens of thousands of scanned images of handwritten digits, together with their correct classifications. MNIST's name comes from the fact that it is a modified subset of two data sets collected by NIST, the United States' National Institute of Standards and Technology. Here's a few images from MNIST:

As you can see, these digits are, in fact, the same as those shown at the beginning of this chapter as a challenge to recognize. Of course, when testing our network we'll ask it to recognize images which aren't in the training set! The MNIST data comes in two parts. The first part contains 60,000 images to be used as training data. These images are scanned handwriting samples from 250 people, half of whom were US Census Bureau employees, and half of whom were high school students. The images are greyscale and 28 by 28 pixels in size. The second part of the MNIST data set is 10,000 images to be used as test data. Again, these are 28 by 28 greyscale images. We'll use the test data to evaluate how well our neural network has learned to recognize digits. To make this a good test of performance, the test data was taken from a set of 250 people than the original training data (albeit still a group split between Census Bureau employees and high school students). This helps give us confidence that our system can recognize digits from people whose writing it didn't see during training. We'll use the notation xx to denote a training input. It'll be convenient to regard each training input xx as a 28×28=78428×28=784-dimensional vector. Each entry in the vector represents the grey value for a single pixel in the image. We'll denote the corresponding desired output by y=y(x)y=y(x), where yy is a 1010-dimensional vector.

For

example,

if

a

particular

training

image, xx,

depicts

a 66,

then y(x)=(0,0,0,0,0,0,1,0,0,0)Ty(x)=(0,0,0,0,0,0,1,0,0,0)T is the desired output from


ARTIFICIAL NEURAL-NETWORKS the network. Note that TT here is the transpose operation, turning a row vector into an ordinary (column) vector. What we'd like is an algorithm which lets us find weights and biases so that the output from the network approximates y(x)y(x) for all training inputs xx. To quantify how well we're achieving this goal we define a cost function.: C(w,b)≡12n∑x∥y(x)−a∥2.(6)(6)C(w,b)≡12n∑x∥y(x)−a∥2. Here, ww denotes the collection of all weights in the network, bb all the biases, nn is the total number of training inputs, aa is the vector of outputs from the network when xx is input, and the sum is over alltraining inputs, xx. Of course, the output aa depends on xx, ww and bb, but to keep the notation simple Ihaven't explicitly indicated this dependence. The notation ∥v∥∥v∥ just denotes the usual length function for a vector vv. We'll call CC the quadratic cost function; it's also sometimes known as the mean squared error or just MSE. Inspecting the form of the quadratic cost function, we see that C(w,b)C(w,b) is non-negative, since every term in the sum is non-negative. Furthermore, the cost C(w,b)C(w,b) becomes small, i.e., C(w,b)≈0C(w,b)≈0, precisely when y(x)y(x) is approximately equal to the output, aa, for all training inputs, xx. So our training algorithm has done a good job if it can find weights and biases so that C(w,b)≈0C(w,b)≈0. By contrast, it's not doing so well when C(w,b)C(w,b) is large - that would mean that y(x)y(x) is not close to the output aa for a large number of inputs. So the aim of our training algorithm will be to minimize the cost C(w,b)C(w,b) as a function of the weights and biases. In other words, we want to find a set of weights and biases which make the cost as small as possible. We'll do that using an algorithm known as gradient descent. Why introduce the quadratic cost? After all, aren't we primarily interested in the number of images correctly classified by the network? Why not try to maximize that number directly, rather than minimizing a proxy measure like the quadratic cost? The problem with that is that the number of images correctly classified is not a smooth function of the weights and biases in the network. For the most part, making small changes to the weights and biases won't cause any change at all in the number of training images classified correctly. That makes it difficult to figure out how to change the weights and biases to get improved performance. If we instead use a smooth cost Department of ECE,MIST Page 28

ARTIFICIAL NEURAL-NETWORKS function like the quadratic cost it turns out to be easy to figure out how to make small changes in the weights and biases so as to get an improvement in the cost. That's why we focus first on minimizing the quadratic cost, and only after that will we examine the classification accuracy. Even given that we want to use a smooth cost function, you may still wonder why we choose the quadratic function used in Equation6. Isn't this a rather ad hoc choice? Perhaps if we chose a different cost function we'd get a totally different set of minimizing weights and biases? This is a valid concern, and later we'll revisit the cost function, and make some modifications. However, the quadratic cost function of Equation6 works perfectly well for understanding the basics of learning in neural networks, so we'll stick with it for now. Recapping, our goal in training a neural network is to find weights and biases which minimize the quadratic cost function C(w,b)C(w,b). This is a well-posed problem, but it's got a lot of distracting structure as currently posed - the interpretation of ww and bb as weights and biases, the σσ function lurking in the background, the choice of network architecture, MNIST, and so on. It turns out that we can understand a tremendous amount by ignoring most of that structure, and just concentrating on the minimization aspect. So for now we're going to forget all about the specific form of the cost function, the connection to neural networks, and so on. Instead, we're going to imagine that we've simply been given a function of many variables and we want to minimize that function. We're going to develop a technique called gradient descent which can be used to solve such minimization problems. Then we'll come back to the specific function we want to minimize for neural networks. Okay, let's suppose we're trying to minimize some function, C(v)C(v). This could be any real-valued function of many variables, v=v1,v2,…v=v1,v2,…. Note that I've replaced the ww and bb notation by vv to emphasize that this could be any function we're not specifically thinking in the neural networks context any more. To minimize C(v)C(v)it helps to imagine CC as a function of just two variables, which we'll call v1v1 and v2v2: What we'd like is to find where CC achieves its global minimum. Now, of course, for the function plotted above, we can eyeball the graph and find the minimum. In that sense, I've perhaps shown slightly too simple a


ARTIFICIAL NEURAL-NETWORKS function! A general function, CC, may be a complicated function of many variables, and it won't usually be possible to just eyeball the graph to find the minimum. One way of attacking the problem is to use calculus to try to find the minimum analytically. We could compute derivatives and then try using them to find places where CC is an extremum. With some luck that might work when CC is a function of just one or a few variables. But it'll turn into a nightmare when we have many more variables. And for neural networks we'll often want far more variables - the biggest neural networks have cost functions which depend on billions of weights and biases in an extremely complicated way. Using calculus to minimize that just won't work!(After asserting that we'll gain insight by imagining CC as a function of just two variables, I've turned around twice in two paragraphs and said, "hey, but what if it's a function of many more than two variables?" Sorry about that. Please believe me when I say that it really does help to imagine CC as a function of two variables. It just happens that sometimes that picture breaks down, and the last two paragraphs were dealing with such breakdowns. Good thinking about mathematics often involves juggling multiple intuitive pictures, learning when it's appropriate to use each picture, and when it's not.) Okay, so calculus doesn't work. Fortunately, there is a beautiful analogy which suggests an algorithm which works pretty well. We start by thinking of our function as a kind of a valley. If you squint just a little at the plot above, that shouldn't be too hard. And we imagine a ball rolling down the slope of the valley. Our everyday experience tells us that the ball will eventually roll to the bottom of the valley. Perhaps we can use this idea as a way to find a minimum for the function? We'd randomly choose a starting point for an (imaginary) ball, and then simulate the motion of the ball as it rolled down to the bottom of the valley. We could do this simulation simply by computing derivatives (and perhaps some second derivatives) of CC - those derivatives would tell us everything we need to know about the local "shape" of the valley, and therefore how our ball should roll. Based on what I've just written, you might suppose that we'll be trying to write down Newton's equations of motion for the ball, considering the effects of friction and gravity, and so on. Actually, we're not going to take the ball-rolling analogy quite that seriously - we're devising an algorithm to minimize CC, not developing an accurate Department of ECE,MIST Page 30

ARTIFICIAL NEURAL-NETWORKS simulation of the laws of physics! The ball's-eye view is meant to stimulate our imagination, not constrain our thinking. So rather than get into all the messy details of physics, let's simply ask ourselves: if we were declared God for a day, and could make up our own laws of physics, dictating to the ball how it should roll, what law or laws of motion could we pick that would make it so the ball always rolled to the bottom of the valley? To make this question more precise, let's think about what happens when we move the ball a small amount Δv1Δv1 in the v1v1 direction, and a small amount Δv2Δv2 in the v2v2 direction. Calculus tells us that CCchanges as follows: ΔC≈∂C∂v1Δv1+∂C∂v2Δv2.(7)(7)ΔC≈∂C∂v1Δv1+∂C∂v2Δv2…(1) . We're going to find a way of choosing Δv1Δv1 and Δv2Δv2 so as to make ΔCΔC negative; i.e., we'll choose them so the ball is rolling down into the valley. To figure out how to make such a choice it helps to defineΔvΔv to be the vector of changes in vv, Δv≡(Δv1,Δv2)TΔv≡(Δv1,Δv2)T, where TT is again the transpose operation, turning row vectors into column vectors. We'll also define the gradient of CC to be the vector of partial derivatives, (∂C∂v1,∂C∂v2)T(∂C∂v1,∂C∂v2)T. We denote the gradient vector by ∇C∇C, i.e.:

∇C≡(∂C/∂v1,∂C/∂v2)T…(2) In a moment we'll rewrite the change ΔCΔC in of ΔvΔv and the gradient, ∇C∇C. Before getting to that, though, I want to clarify something that sometimes gets people hung up on the gradient. When meeting the ∇C∇C notation for the first time, people sometimes wonder how they should think about the ∇∇ symbol. What, exactly, does ∇∇ mean? In fact, it's perfectly fine to think of ∇C∇C as a single mathematical object - the vector defined above - which happens to be written using two symbols. In this point of view, ∇∇ is just a piece of notational flag-waving, telling you "hey, ∇C∇C is a gradient vector". There are more advanced points of view where ∇∇ can be viewed as an independent mathematical entity in its own right (for example, as a differential operator), but we won't need such points of view. With these definitions, the expression7 for ΔCΔC can be rewritten as ΔC≈∇C⋅Δv.(9)(9)ΔC≈∇C⋅Δv…(3) This equation helps explain why ∇C∇C is called the gradient vector: ∇C∇C relates changes in vv to changes in CC, just as we'd expect something Department of ECE,MIST Page 31

ARTIFICIAL NEURAL-NETWORKS called a gradient to do. But what's really exciting about the equation is that it lets us see how to choose ΔvΔv so as to make ΔCΔC negative. In particular, suppose we choose Δv=−η∇C,(10)(10)Δv=−η∇C…(4) where ηη is a small, positive parameter (known as the learning rate). Then Equation [9] tells

us

that

ΔC≈−η∇C⋅∇C=−η∥∇C∥2ΔC≈−η∇C⋅∇C=−η∥∇C∥2

Because ∥∇C∥2≥0∥∇C∥2≥0, this guarantees that ΔC≤0ΔC≤0, i.e., CC will always decrease, never increase,

if we change vv according to the prescription in 10.

(Within, of course, the limits of the approximation in Equation9. This is exactly the property we wanted!

And so we'll take Equation

10 to define the "law of motion"

for the ball in our gradient descent algorithm.

That is, we'll use Equation10 to compute a value for ΔvΔv, then move the ball's position vv by that amount: v→v′=v−η∇C.(11)(11)v→v′=v−η∇C…(5) Then we'll use this update rule again, to make another move. If we keep doing this, over and over, we'll keep decreasing CC until - we hope - we reach a global minimum.

I've explained gradient descent when CC is a function of just two

variables. But, in fact, everything works just as well even when CC is a function of many more variables. Suppose in particular that CC is a function of mm variables, v1, …,vmv1,…,vm. Then the change ΔCΔC in Croduced by a small change Δv=(Δv1, …,Δvm)TΔv=(Δv1,…,Δvm)T is ΔC≈∇C⋅Δv,(12)(12)ΔC≈∇C⋅Δv

…(6)

where the gradient ∇C∇C is the vector ∇C≡(∂C∂v1,…,∂C∂vm)T.(13)(13)∇C≡(∂C∂v1,…,∂C∂vm)T…(7) Just as for the two variable case, we can choose Δv=−η∇C,(14)(14)Δv=−η∇C

..(8) Department of

ECE,MIST Page 32


and we're guaranteed that our (approximate) expression12 for ΔCΔCwill be negative. This gives us a way of following the gradient to a minimum, even when CC is a function of many variables, by repeatedly applying the update rule v→v′=v−η∇C.(15)(15)v→v′=v−η∇C. You can think of this update rule as defining the gradient descent algorithm. It gives us a way of repeatedly changing the position vv in order to find a minimum of the function CC. The rule doesn't always work - several things can go wrong and prevent gradient descent from finding the global minimum of CC, a point we'll return to explore in later chapters. But, in practice gradient descent often works extremely well, and in neural networks we'll find that it's a powerful way of minimizing the cost function, and so helping the net learn.

While our neural network gives impressive performance, that performance is somewhat mysterious. The weights and biases in the network were discovered automatically. And that means we don't immediately have an explanation of how the network does what it does. Can we find some way to understand the principles by which our network is classifying handwritten digits? And, given such principles, can we do better. To put these questions more starkly, suppose that a few decades hence neural networks lead to artificial intelligence (AI). Will we understand how such intelligent networks work? Perhaps the networks will be opaque to us, with weights and biases we don't understand, because they've been learned automatically. In the early days of AI research people hoped that the effort to build an AI would also help us understand the principles behind intelligence and, maybe, the functioning of the human brain. But perhaps the outcome will be that we end up understanding neither the brain nor how artificial intelligence works. To address these questions, let's think back to the interpretation of artificial neurons that I gave at the start of the chapter, as a means of weighing evidence. Suppose we want to determine whether an image shows a human face or not:



Fig 18: Face Recogniton We could attack this problem the same way we attacked handwriting recognition by using the pixels in the image as input to a neural network, with the output from the network a single neuron indicating either "Yes, it's a face" or "No, it's not a face".

Let's suppose we do this, but that we're not using a learning algorithm. Instead, we're going to try to design a network by hand, choosing appropriate weights and biases. How might we go about it? Forgetting neural networks entirely for the moment, a heuristic we could use is to decompose the problem into sub-problems: does the image have an eye in the top left? Does it have an eye in the top right? Does it have a nose in the middle? Does it have a mouth in the bottom middle? Is there hair on top? And so on. If the answers to several of these questions are "yes", or even just "probably yes", then we'd conclude that the image is likely to be a face. Conversely, if the answers to most of the questions are "no", then the image probably isn't a face. Of course, this is just a rough heuristic, and it suffers from many deficiencies. Maybe the person is bald, so they have no hair. Maybe we can only see part of the face, or the face is at an angle, so some of the facial features are obscured. Still, the heuristic suggests that if we can solve the sub-problems using neural networks, then perhaps we can build a neural network for face-detection, by combining the networks for the sub-problems. Here's a possible architecture, with rectangles denoting the subDepartment of ECE,MIST Page 34

ARTIFICIAL NEURAL-NETWORKS networks. Note that this isn't intended as a realistic approach to solving the facedetection problem; rather, it's to help us build intuition about how networks function. Here's the architecture:

Fig 19: Architecture of Face Detection

It's also plausible that the sub-networks can be decomposed. Suppose we're considering the question: "Is there an eye in the top left?" This can be decomposed into questions such as: "Is there an eyebrow?"; "Are there eyelashes?"; "Is there an iris?"; and so on. Of course, these questions should really include positional information, as well - "Is the eyebrow in the top left, and above the iris?", that kind of thing - but let's keep it simple. The network to answer the question "Is there an eye in the top left?" can now be decomposed:

Fig 20: Decomposition of Network Those questions too can be broken down, further and further through multiple layers. Ultimately, we'll be working with sub-networks that answer questions so simple they can easily be answered at the level of single pixels. Those questions might, for example, be about the presence or absence of very simple shapes at


ARTIFICIAL NEURAL-NETWORKS particular points in the image. Such questions can be answered by single neurons connected to the raw pixels in the image. The end result is a network which breaks down a very complicated question - does this image show a face or not - into very simple questions answerable at the level of single pixels. It does this through a series of many layers, with early layers answering very simple and specific questions about the input image, and later layers building up a hierarchy of ever more complex and abstract concepts. Networks with this kind of many-layer structure - two or more hidden layers - are called deep neural networks. Of course, I haven't said how to do this recursive decomposition into sub-networks. It certainly isn't practical to hand-design the weights and biases in the network. Instead, we'd like to use learning algorithms so that the network can automatically learn the weights and biases - and thus, the hierarchy of concepts - from training data.

Researchers in the 1980s and 1990s tried using stochastic gradient descent and backpropagation to train deep networks. Unfortunately, except for a few special architectures, they didn't have much luck. The networks would learn, but very slowly, and in practice often too slowly to be useful. Since 2006, a set of techniques has been developed that enable learning in deep neural nets. These deep learning techniques are based on stochastic gradient descent and backpropagation, but also introduce new ideas. These techniques have enabled much deeper (and larger) networks to be trained - people now routinely train networks with 5 to 10 hidden layers. And, it turns out that these perform far better on many problems than shallow neural networks, i.e., networks with just a single hidden layer. The reason, of course, is the ability of deep nets to build up a complex hierarchy of concepts. It's a bit like the way conventional programming languages use modular design and ideas about abstraction to enable the creation of complex computer programs. Comparing a deep network to a shallow network is a bit like comparing a programming language with the ability to make function calls to a stripped down language with no ability to make such calls. Abstraction takes a different form in neural networks than it does in conventional programming, but it's just as important Department of ECE,MIST Page 36


Recent advances and future applications of NNs include: 7.5 Integration of fuzzy logic into neural networks Fuzzy logic is a type of logic that recognizes more than simple true and false values, hence better simulating the real world. For example, the statement today is sunny might be 100% true if there are no clouds, 80% true if there are a few clouds, 50% true if it's hazy, and 0% true if rains all day. Hence, it takes into concepts like -usually, somewhat, and sometimes. Fuzzy logic and neural networks have been integrated for uses as diverse as automotive engineering, applicant screening for jobs, the control of a crane, and the monitoring of glaucoma.

7.6 Pulsed neural networks "Most practical applications of artificial neural networks are based on a computational model involving the propagation of continuous variables from one processing unit to the next. In recent years, data from neurobiological experiments have made it increasingly clear that biological neural networks, which communicate through pulses, use the timing of the pulses to transmit information and perform Department of ECE,MIST Page 37

ARTIFICIAL NEURAL-NETWORKS computation. This realization has stimulated significant research on pulsed neural networks, includingtheoretical analyses and model development, neurobiological modeling, and hardware implementation."

7.7 Hardware specialized for neural networks Some networks have been hardcoded into chips or analog devices ? this technology will become more useful as the networks we use become more complex.The primary benefit of directly encoding neural networks onto chips or specialized analog devices is SPEED!NN hardware currently runs in a few niche areas, such as those areas where very high performance is required (e.g. high energy physics) and in embedded applications of simple, hardwired networks (e.g. voice recognition).

Many NNs today use less than 100 neurons and only need occasional training. In these situations, software simulation is usually found sufficientWhen NN algorithms develop to the point where useful things can be done with 1000's of neurons and 10000's of synapses, high performance NN hardware will become essential for practical operation



CHAPTER 8 CONCLUSION The computing world has a lot to gain fron neural networks. Their ability to learn by example makes them very flexible and powerful. Furthermore there is no need to devise an algorithm in order to perform a specific task; i.e. there is no need to understand the internal mechanisms of that task. They are also very well suited for real time systems because of their fast responseand computational times which are due to their parallel architecture. Neural networks also contribute to other areas of research such as neurology and psychology. They are regularly used to model parts of living organisms and to investigate the internal mechanisms of the brain. Perhaps the most exciting aspect of neural networks is the possibility that some day 'consious' networks might be produced. There is a number of scientists arguing that conciousness is a 'mechanical' property and that 'consious' neural networks are a realistic possibility. Department of ECE,MIST Page 39

ARTIFICIAL NEURAL-NETWORKS Finally, I would like to state that even though neural networks have a huge potential we will only get the best of them when they are intergrated with computing, AI, fuzzy logic and related subjects.

.

CHAPTER 9 FUTURE SCOPE 

Improved stock prediction



Common usage of self-driven cars



Self –Diagnosis of medical problems



Auto Transfer of hand written documents into formatted word processing documents



REFERENCE 1)

Web-Site

made

by

Christos

Stergiou

and

Dimitrios

Siganos

http://www.doc.ic.ac.uk 2) S.A. Shumsky “Selected lections about neural computing” 3) Simon Haykin “Kalman Filtering and Neural Networks” Copyright © 2001 John Wiley & Sons, Inc. ISBNs: 0-471-36998-5 (Hardback); 0-471-22154-6 (Electronic) 4) Web-Site made by BaseGroup Labs © 1995-2006 http://www.basegroup.ru 5) Vesselin Vatchev, USC November 20, 2002, “The analysis of the Empirical Mode Decomposition Method ”


Neural Network Doc 1y1m4d

Overview 3e4r5l

More details w3441

Related Documents 3m3m1z

Neural Network Doc 1y1m4d

Neural Network 5q5738

Artificial Neural Network ,.ppt 58455a

Neural Network Ppt Presentation 3z1c2j

Neural Network Using Matlab 3t3b6f

Probabilistic Neural Network (pnn) 2i1h5q

More Documents from "anil kumar" 676a2t

Seat Matrix 2017 vx2h

Anurcet 2x2m4

6l6m1j

Gjl 300 3y5x1p

Neural Network Doc 1y1m4d

61657786 Bts Site Audit Checklist 54j1o