A neural network is better viewed as a collection of multiple optimisation processes, each with its own internal memory.