Based on this performance measure, RNN weights are altered in the direction that is most likely to reduce the RNN error E_t(t). This alteration is driven by a derivative-based algorithm. The error gradient is derived from Equation (11.6). The error gradient are computed according to

where   d(t) = desired (target) output.

Ordered derivatives are a special case of the chain rule only applicable for an ordered system where the values can be computed one by one in the order of E₁, E₂, … E_i. By using the definitions of ordered derivatives presented by Werbos [6], the error derivatives for each neuron are computed according to

where

The ordered derivatives for the weights can be calculated as

And, the weights are updated according to

where ε is a learning rate, which is used for RNN learning speed adjustments. The larger the value of the learning rate, the faster adaptation occurs. However, if the learning rate is too large, the RNN can diverge, leading to increasing errors and erroneous results. On the other hand, the smaller the value of the learning rate, the slower adaptation tends to be, but the more likely the RNN is to converge.

In RNNs, Equations 11.2 to 11.12 are applied repeatedly until the RNN can produce the desired outputs within a given tolerance. This tolerance is set by the RNN user.

As mentioned earlier in this section, different NNs have very different structures and distinguishing characteristics. Therefore, NN users must select a NN that is most suitable to the particular problem they wish to solve.

BORN is an RNN connection optimization algorithm, which can eliminate unnecessary RNN connections. A brief explanation is provided below. In order to use the BORN algorithm in RNNs with BP learning, a new function g(C_ij) must be introduced into the RNN connections. This function operates on the connections between neurons. If g(C_ij) = 1.0, a connection between the i^th and j^th neurons is present. If, however, g(C_ij) = 0.0, the connection is not present. In order to apply the BORN algorithm in RNNs, Equation 11.3 must be changed to

where

where α is a constant, typically set to a value greater than 10.0. For sufficiently large α, g(C_ij) ≅ 1.0 for C_ij > 0, and g(C_ij) ≅ 0.0 for C_ij < 0.

If the connection strength, C_ij, is set to 0.0 at the beginning of the training, and if BP is applied to update C_ij, g(C_ij) will approach either 1.0 or 0.0. Now, the error derivative can be expressed as

For the RNN, the error gradients for the weights are expressed as

The weight and connection updates are expressed as

For RNN training, Equations 11.13 to 11.18 are applied repeatedly until the RNN can produce the desired outputs within some pre-defined tolerance on the RNN error. Further details of the BORN algorithm and example applications can be found in Reference [12].

NN CONNECTIVITIES OPTIMIZATION WITH GA

This section performs BP Neural Network weight selection and connectivity optimizations by GA and compares the results with BORN algorithm results. The purpose of this study is to compare the stability and robustness of the BORN algorithm and GA. Simple GA [2] was used to verify the performance of the Neural Network with GA optimization.

Decode and Fitness Function for GA-NN

In order to optimize NN connections by GA, a decode routine and a fitness function must be developed. The number of NN connections is defined as the sum of the number of NN inputs, hidden neurons, and outputs. Since, in this chapter, the NN connections are represented in binary form (such as connected is I and disconnected is 0), this can be translated to GA as a 1-bit multi-parameter problem. Therefore, the decode function is applied such as “if an allele is 1, then the NN connection exists; otherwise no connection” for the number of connections, which is essentially the same as the chromosome length. Listing 11.1 is the complete C language listing for decode function used in this study. The objective of the GA is to maximize the fitness function, f, defined in Equation 11.19.

void decode(chromosome *chrom, int *lchrom)
{
int i, j, counter;
counter=0;
for(i=(NN_Num_Input+ NN_Num_Hidden+ NN_Num_Output);i>=1;--i){
for(j=( NN_Num_Input+ NN_Num_Hidden+ NN_Num_Output);j>=1;--j){
if( chrom[counter] ){ ConnectionMatrix[i][j] = 1;} //Connection Exists
else { ConnectionMatrix[i][j] = 0; } //No Connection
++counter;}}
return;
}

Listing 11.1 Decode function for GA-NN connection optimization problem.