The hidden layer
The hidden layer contains a multiplication function of weights and an addition of biases, as shown in the following diagram:
A dataflow graph with multiplication function
The input is sent to the multiplication function along with the weights. Thus x1 and x2 will be multiplied by the weights as explained in Chapter 4, Become an Unconventional Innovator. In this model, the bias variable is added to the result of the multiplications to each of the hidden neurons (one for each x). Go back to Chapter 4, Become an Unconventional Innovator, again to see the low-level mechanics of the system if necessary.
Bear in mind that there are several mathematical ways, and thus there are several architectures represented in graphs that can solve the FNN XOR problem. Once the multiplication and addition have been computed, a logistic sigmoid function (see Chapter 2, Think like a Machine) is applied to the result.
Why is a logistic sigmoid function needed here?
In Chapter 4, Become an Unconventional Innovator, in my vintage program, I did not use one. I just rounded the values to 0 or 1. That works in a limited environment. However, as explained in Chapter 2, Think like a Machine, the output of the neurons needs to be normalized to be useful when the input data starts increasing and persifying. To sum it up, suppose we obtain 200 as an input of a neuron and 2 as an output of another neuron. That makes it very difficult to manage this in the next layer. It's as if you went to a shop and had to pay for one product in a given currency and another product in a different currency. Now the cashier has to add that up. It's not practical at all in a shop and even less in a network with thousands of neurons.
The following LS logistic sigmoid is thus a very handy tool for the outputs to level the values:
LS = tf.sigmoid(tf.matmul(x_, W1) + B1)
Basically, the values are squashed into small numbers that are compatible with the next step, which is the computation of the output layer.