Deep Learning

Deep Learning – Basics of Recursive Neural Network

Recursive Neural Network – Recurrent neural networks are in fact Recursive neural networks. In other words recursive neural network is just a generalisation of a recurrent neural network. In a recurrent network the weights are shared along the length of the sequence though dimensionality remains constant. Answer to why this way is simple because it helps to deal with position-dependent weights when encounter a sequence at test-time of different length at train-time.

 

What is Deep Learning?

“Deep learning is undeniably mind-blowing” machine learning technique that teaches computers to do what comes naturally to humans: learn by example. It can be used with ease to predict the unpredictable”. Researchers and engineers are busy in creating artificial intelligence by using a combination of non-bio-neural networks and natural intelligence”.

Deep Learning, in short, is going much beyond machine learning and its algorithms that are either supervised or unsupervised. In DL it uses many layers of nonlinear processing units for feature extraction and transformation. It has revolutionised today’s industries by demonstrating near human-level accuracy in certain tasks. The task like pattern recognition, image classification, voice/text decoding and many more.

Deep Leaning is a key technology

  • To voice control in mobile devices like handphones, TVs, vice command enabled speakers and TVs
  • Behind driverless cars, enabling them to recognise a stop sign or to distinguish a pedestrian from a lamppost.
  • Has revolutionised, image processing & classification and also speech recognition with high accuracy.

Deep learning is getting lots of attention lately and for good reason. It is achieving results that were not possible before. Business leaders and developers community absolutely need to understand what it is, what it can do and how it works.

 

What is Recursive Neural Network?

Recursive neural network are family member and a kind of deep neural network. They are generally created after applying the same set of weights recursively on the structured inputs. This happens at at every node for the same reason. RNNS are comprise of a class of architecture that operates on structured inputs, and in particular, on directed acyclic graphs.

Recursive Neural Networks – Call it as a deep tree-like structure. When the need is to parse a whole sentence we use a recursive neural network. Tree-like topology allows branching connections and hierarchical structure. Arguments here can be how recursive neural networks are different than recurrent neural networks? We can simply do the classification of recursive neural network approaches into two as below

IRNN

  • Inner Approach – This approach is usually conduct recursion inside the underlying graph and objective is achieved usually by moving forward slowly around edges of the graph.
  • Outer Approach – This approach is usually conduct recursion by outside the underlying graph and aggregate information over progressively longer distances in a rectangular direction.

RNNs are used to predict structured outputs over variable-size input structures, some time a scalar prediction as well by traversing a given structure in topological order. Recursive neural networks respond not only to input, but  to context as well. They process each input of time series separately. The first introduction of RNNs happened just to meet the need of learning distributed representations of structure, such as logical terms.

Recurrent neural networks are recursive artificial neural networks with a certain structure, that of a linear chain. Whereas recursive neural networks operate on any hierarchical structure, combining child representations into parent representations. recurrent neural networks operate on the linear progression of time, combining the previous time step and a hidden representation into the representation for the current time step [As per wiki]

  • Questions – How recursive neural networks are different than recurrent neural networks?
  • Answer – Recurrent neural networks are in fact recursive neural networks with a particular structure: that of a linear chain.

RNNs are hierarchical kind of network with no time aspect to the input sequence but the input has to be processed hierarchically in a tree fashion.

 

Recurrent vs Recursive Neural Networks

As per  Wikipedia, recurrent neural networks are in fact recursive neural networks. Both are usually denoted by the same acronym: RNN. In short recurrent neural networks are recurring over time i.e. recursive network is just a generalisation of a recurrent network.

ORNNIt is quite simple to see from the picture why it is called a recursive neural network. Each parent node’s children are simply a node similar to that node. So its quite evident it is more like a hierarchical network where there is really no time aspect to the input sequence but the input has to be processed hierarchically in a tree fashion. Recursive neural networks operate on any hierarchical structure, combining child representations into parent representations.

In general  difference between recursive and recurrent networks is not well defined. The efficiency of a recursive network is better than a feed forward network. Recurrent neural networks are generally chain like structure as they really don’t branch but for recurrent they are more of deep tree structure. Recurrent networks has difficulty in dealing with tree like structure which is not for recurrent. When you parse a sentence (NLP processing) the easy way is apply tree-like topology which does branching of connections.

As now we know networks operate on structured class are more of recursive networks. If we stack multiple recursive layers then those can be called as deep recursive neural network.

 

Not Covered here

Topics we have not covered in this post but are extremely critical and important to understand in order to get little more strong hands-on RNNs as below.

  • Sequential Memory
  • Backpropagation
  • LSTM’s and GRU’s

 

Points to Note:

All credits if any remains on the original contributor only. We have covered all basics around Recursive Neural Network. RNNs are all about modelling units in sequence. The perfect support for Natural Language Processing – NLP tasks. Though often such tasks struggle to find best companion between CNN’s and RNNs algorithms to look for information.

 

Books + Other readings Referred

  • Research through open internet, news portals, white papers and imparted knowledge via live conferences & lectures.
  • Lab and hands-on experience of  @AILabPage (Self-taught learners group) members.
  • This useful pdf on NLP parsing with Recursive NN.
  • Amazing information in this pdf as well.

 

Machine Learning (ML) - Everything You Need To KnowConclusion –  I particularly think that getting to know the types of machine learning algorithm actually helps to see a somewhat clear picture. The answer to the question “What machine learning algorithm should I use?” is always “It depends.” It depends on the size, quality, and nature of the data. Also, what is the objective/motive data torturing? As more we torture data more useful information comes out. It depends on how the math of the algorithm was translated into instructions for the computer you are using. And it depends on how much time you have. To us, at AILabPage we say machine learning is crystal clear and simple task. It is not only for PhDs aspirants but it’s for you, us and everyone.

 

======================= About the Author =======================

Read about Author at : About Me

Thank you all, for spending your time reading this post. Please share your opinion / comments / critics / agreements or disagreement. Remark for more details about posts, subjects and relevance please read the disclaimer.

FacebookPage    ContactMe      Twitter

============================================================

 

 

Facebook Comments
Advertisements

1 reply »

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.