The Attention Mechanism from Scratch

Tweet Share Share Last Updated on September 20, 2021 The attention mechanism was introduced to improve the performance of the encoder-decoder model for machine translation. The idea behind the attention mechanism was to permit the decoder to utilize the most relevant parts of the input sequence in...

11h

A Gentle Introduction to Particle Swarm Optimization

Tweet Share Share Particle swarm optimization (PSO) is one of the bio-inspired algorithms and it is a simple one to search for an optimal solution in the solution space. It is different from other optimization algorithms in such a way that only the objective function is needed and it is not dependent...

11h

What is Attention?

Tweet Share Share Last Updated on September 15, 2021 Attention is becoming increasingly popular in machine learning, but what makes it such an attractive concept? What is the relationship between attention as applied in artificial neural networks, and its biological counterpart? What are the components...

1w

A Bird’s Eye View of Research on Attention

Tweet Share Share Last Updated on September 9, 2021 Attention is a concept that is scientifically studied across multiple disciplines, including psychology, neuroscience and, more recently, machine learning. While all disciplines may have produced their own definitions for attention, there is one core...

2w

Lagrange Multiplier Approach with Inequality Constraints

Tweet Share Share Last Updated on August 28, 2021 In a previous post, we introduced the method of Lagrange multipliers to find local minima or local maxima of a function with equality constraints. The same method can be applied to those with inequality constraints as well. In this tutorial, you will...

4w

A Gentle Introduction To Approximation

Tweet Share Share When it comes to machine learning tasks such as classification or regression, approximation techniques play a key role in learning from the data. Many machine learning methods approximate a function or a mapping between the inputs and outputs via a learning algorithm. In this tutorial,...

4w

A Gentle Introduction To Sigmoid Function

Tweet Share Share Whether you implement a neural network yourself or you use a built in library for neural network learning, it is of paramount importance to understand the significance of a sigmoid function. The sigmoid function is the key to understanding how a neural network learns complex problems....

4w

Calculus in Action: Neural Networks

Tweet Share Share An artificial neural network is a computational model that approximates a mapping between inputs and outputs.  It is inspired by the structure of the human brain, in that it is similarly composed of a network of interconnected neurons that propagate information upon receiving sets...

4w

A Gentle Introduction to Taylor Series

Tweet Share Share A Gentle Introduction to Taylor Series Taylor series expansion is an awesome concept, not only the world of mathematics, but also in optimization theory, function approximation and machine learning. It is widely applied in numerical computations when estimates of a function’s values...

4w

A Gentle Introduction to Optimization / Mathematical Programming

Tweet Share Share Last Updated on August 10, 2021 Whether it is a supervised learning problem or an unsupervised problem, there will be some optimization algorithm working in the background. Almost any classification, regression or clustering problem can be cast as an optimization problem. In this...

Aug 2021