What are recurrent neural networks?

Recurrent neural networks use feedback to create a kind of memory. They are primarily used when working with se­quen­tial data and require a re­l­at­ively high level of training.

What is a recurrent neural network?

A recurrent neural network (RNN) is an ar­ti­fi­cial neural network that can create feedback and thus feed output in­form­a­tion back in as input. The method is used fre­quently in pro­cessing se­quen­tial data for deep learning and AI. In taking in in­form­a­tion from previous inputs, the recurrent neural network develops a kind of memory and changes the output based on previous elements in the sequence. There are different types of feedback, which provide increased pos­sib­il­it­ies but also require more training.

How do recurrent neural networks work?

At first glance, recurrent neural networks are built like other neural networks. They consist of at least three different layers, that in turn contain neurons (nodes), which are connected to each other. There is an input layer, an output layer and any number of hidden layers.

• Input layer: This layer is where in­form­a­tion from the outside world is received. That in­form­a­tion is then weighted and passed on to the next layer.
• Hidden layer(s): The in­form­a­tion is passed on and weighted again in the hidden layer. That happens con­tinu­ously as the in­form­a­tion moves through the layers. It takes place hidden from view, which is where the layer gets its name from.
• Output layer: The weighted and modified in­form­a­tion is then output. That output is either the end result or serves as the input for other neurons.

Close

What makes recurrent neural networks special is the ability to determine the direction of in­form­a­tion flow yourself. Thanks to the feedback, a neuron’s output doesn’t ne­ces­sar­ily need to get sent to the next higher layer. It can also be used as input for the same neuron, a neuron in the same layer or a neuron in a previous layer. The weighting parameter is the same in each layer. Feedback is divided into four different cat­egor­ies:

• Direct feedback: The output is used as input for the same neuron.
• Indirect feedback: The output is linked to a previous layer.
• Lateral feedback: The output is connected to the input of another neuron in the same layer.
• Full feedback: All output from neurons is connected to other neurons.

How are recurrent neural networks trained?

Training recurrent neural networks is very com­plic­ated, since in­form­a­tion from the distant past cannot be suf­fi­ciently taken into account, if at all. Functions like Long Short-Term Memory (LSTM) are often used to solve that problem, allowing in­form­a­tion to be stored for longer. With the right training, recurrent neural networks can dis­tin­guish important from un­im­port­ant in­form­a­tion and feed the former back into the network.

What kinds of ar­chi­tec­tures are there?

There are various kinds of ar­chi­tec­tures for RNNs:

• One to many: The RNN receives one input and returns multiple outputs. An example is an image caption, where a sentence is generated from a keyword.
• Many to one: Several inputs are assigned to a single output. This type of ar­chi­tec­ture is used in sentiment analysis, for example, where feedback is predicted based on ref­er­ences.
• Many to many: Several inputs are used to get several outputs. There don’t have to be the same number of inputs and outputs. This type is used when trans­lat­ing one language into another.

What are the most important ap­plic­a­tions of recurrent neural networks?

Recurrent neural networks are es­pe­cially useful when context is important for in­form­a­tion pro­cessing. This is es­pe­cially true for se­quen­tial data, which have a set order or temporal sequence. In those cases, RNNs can do a good job of pre­dict­ing a result based on the context of previous inputs. That makes them suitable for areas like the following:

• Forecasts and time series data: RNNs are often used to predict events or values based on his­tor­ic­al data, as in financial market analysis, weather forecasts and energy use.
• Natural language pro­cessing (NLP): RNNs are es­pe­cially useful in natural language pro­cessing. They’re used for machine trans­la­tion, language modelling, text re­cog­ni­tion and automatic text gen­er­a­tion.
• Speech re­cog­ni­tion: RNNs play a key role in speech re­cog­ni­tion tech­no­lo­gies, where they’re used to convert spoken language into text. That tech­no­logy is used in digital as­sist­ants and automatic tran­scrip­tion services.
• Image and video analysis: While RNNs are primarily used for pro­cessing se­quen­tial data, they can also be used in image and video analysis. That’s es­pe­cially true when it comes to pro­cessing image sequences and videos, as is done for video summaries and detecting activity on security cameras.

What is the dif­fer­ence between RNNs and feed­for­ward neural networks?

Recurrent neural networks are a step further than feed­for­ward neural networks (FNN), which don’t allow feedback. In FNNs, in­form­a­tion flows in only one direction, to the next highest layer. Inputting in­form­a­tion backwards is thus not possible. Although FNNs can recognise patterns, they cannot access in­form­a­tion that has already been processed.