in Education by
I am having problem in understanding the difference between three optimizers for loss.I went through some documents to understand the principles but I want to know, when is it preferable to use one in place of others?What are the important differences? Select the correct answer from above options

1 Answer

0 votes
by
This answer will give you a brief explanation: 1.Momentum: It helps SGD to navigate along with relevant directions and softens oscillation in the irrelevant directions.It adds a fraction of the direction of the previous step to that of the current step which increases the speed of amplification in the correct direction, the fraction is usually in the range of 0 to 1. There is one major disadvantage with momentum which is when we are close to the goal, the momentum is usually very high and doesn’t slow down which causes it to miss or oscillate in and around the minima. 2. Nesterov accelerated gradient: It solves the disadvantage of momentum by starting to slow down early. Nag performs the same thing as momentum but in some other way,first it makes a big jump based on all the previous information, then calculates the gradient and makes some small changes. These changes give significant practical speedups. 3.AdaGrad allows the learning to adapt based on different parameters. It performs small updates for frequent parameters and large updates for infrequent parameters.It also eliminates the need for tuning the learning rate. Here, each parameter have its own learning rate and it decreases monotonically due to the peculiarities of the algorithm. 4.AdaDelta It resolves the monotonically decreasing problem of AdaGrad.In AdaDelta it used a sliding window which allows the sum to decrease instead of summing all the past square roots.RMSprop is very similar to AdaDelta. 5. Adam has an algorithm similar to AdaDelta.It stores momentum changes along with learning rates for each of the parameters. Go through this TensorFlow Tutorial to get a clear understanding of TensorFlow!

Related questions

0 votes
    I was looking at the docs of TensorFlow about tf.nn.conv2d here. But I can't understand what it does or ... it raised the question. Select the correct answer from above options...
asked Jan 27, 2022 in Education by JackTerrance
0 votes
    In tf.nn.max_pool of tensorflow what is the difference between 'SAME' and 'VALID'? I read in here that ... pool means in tensorflow? Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    I read that regularization terms are implemented by manually adding an additional term to loss value in neural network ... it manually? Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    What is global_step ? why do we use '0' while setting up global_step? def training(loss,learning_rate): tf. ... global_step becomes 1? Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    I want to assign a value to a TensorFlow variable in Python and I am using this: import tensorflow as tf import ... do to correct this? Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    I want different learning layers in different layers just like we do in Caffe. I just want to speed up the training ... can I do this? Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    I'm trying to implement stochastic gradient descent in MATLAB however I am not seeing any convergence. Mini-batch ... cost of 420. Select the correct answer from above options...
asked Feb 1, 2022 in Education by JackTerrance
0 votes
    I am trying to understand the role of the Flatten function in Keras. Below is my code, which is a simple two ... flatten it? Thanks! Select the correct answer from above options...
asked Feb 8, 2022 in Education by JackTerrance
0 votes
    I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification ... labels? Select the correct answer from above options...
asked Jan 31, 2022 in Education by JackTerrance
0 votes
    What is the role of Flatten in Keras. I am executing the code below and it's a two layered network. The ... output is already flat? Select the correct answer from above options...
asked Jan 25, 2022 in Education by JackTerrance
0 votes
    I want my model to be trained with some classified images. They are of different sizes, so how can I train ... resizing the images? Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    I am searching for the better option to save a trained model in PyTorch, these are the options I am using ... the above two functions? Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    What does the following function do? Should I consider it as a lookup table like in skip-gram model? tf.nn. ... mod', name=None) Select the correct answer from above options...
asked Jan 24, 2022 in Education by JackTerrance
0 votes
    Using up-to-date Keras and Tensorflow on Ubuntu 16.04 and 14.04. For the following code: img2D = ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 23, 2022 in Education by JackTerrance
0 votes
    Suppose I have a Tensorflow tensor. How do I get the dimensions (shape) of the tensor as integer values? I ... 'Dimension' instead. Select the correct answer from above options...
asked Feb 8, 2022 in Education by JackTerrance
...