Gradient Descent gaunthan Posted on Oct 14 2017 ? Machine Learning ? ## Definition ## Tips ### Feature scaling Involves dividing the input values by the range (i.e. the maximum value minus the minimum value) of the input variable, resulting in a new range of just 1. $$-0.5≤x_{(i)}≤0.5$$ ### Mean normalization Involves subtracting the average value for an input variable from the values for that input variable resulting in a new average value for the input variable of just zero. $$x_i := \dfrac{x_i - \mu_i}{s_i}$$ Where $μ_i$ is the average of all the values for feature (i) and $s_i$ is the range of values (max - min), or $s_i$ is the standard deviation. For example, if $x_i$ represents housing prices with a range of 100 to 2000 and a mean value of 1000, then, $$x_i := \dfrac{price-1000}{1900}$$ ### Learning Rate α - If α is too small: slow convergence - If α is too large: J(θ) may not decrease on every iteration; may not converge. To choose α, try $$ ..., 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, ...$$ ### Debugging gradient descent Make a plot with number of iterations on the x-axis. Now plot the cost function, J(θ) over the number of iterations of gradient descent. If J(θ) ever increases, then you probably need to decrease α. ### Automatic convergence test Declare convergence if J(θ) decreases by less than E in one iteration, where E is some small value such as $10^{−3}$. However in practice it's difficult to choose this threshold value. ### Improve Features We can improve our features and the form of our hypothesis function in a couple different ways. We can **combine** multiple features into one. For example, we can combine $x_1$ and $x_2$ into a new feature $x_3$ by taking $x_1⋅x_2$. ### Polynomial Regression Our hypothesis function need not be linear (a straight line) if that does not fit the data well. We can change the behavior or curve of our hypothesis function by making it a quadratic, cubic or square root function (or any other form). 赏 Wechat Pay Alipay Normal Equation The Basic Concepts of Machine Learning
