Question

Please (a) Compute the first-order derivative ∇E(w). You will need to provide the intermediate steps of derivation. (b) Once the optimal w is obtain, it will be used to make predictions as follows: Predicted class of x={1−1​ if θ(wTx)≥0.5 if θ(wTx)<0.5​ where the function θ(z)=1+e−z1​ looks like Explain why the decision boundary of logistic regression is still linear, though the linear signal wTx is passed through a nonlinear function θ to compute the outcome of prediction. (c) Is the decision boundary still linear if the prediction rule is changed to the following? Justify briefly. Predicted class of x={1−1​ if θ(wTx)≥0.9 if θ(wTx)<0.9​ (d) In light of your answers to the above two questions, what is the essential property of logistic regression that results in the linear decision boundary? E(w)=N1​∑n=1N​ln(1+e−yn​wTxn​). Please (a) Compute the first-order derivative ∇E(w). You will need to provide the intermediate steps of derivation. (b) Once the optimal w is obtain, it will be used to make predictions as follows: Predicted class of x={1−1​ if θ(wTx)≥0.5 if θ(wTx)<0.5​ where the function θ(z)=1+e−x1​ looks like Explain why the decision boundary of logistic regression is still linear, though the linear signal wTx is passed through a nonlinear function θ to compute the outcome of prediction. (c) Is the decision boundary still linear if the prediction rule is changed to the following? Justify briefly. Predicted class of x={1−1​ if θ(wTx)≥0.9 if θ(wTx)<0.9​

Answer 1

The first-order derivative of E(w) is computed using the chain rule, and the logistic regression maintains a linear decision boundary because classification is based on whether wTx crosses a threshold, irrespective of the value of the threshold. Changing the threshold value shifts the boundary but does not affect its linearity.

Step-by-step explanation:

The calculation of the first-order derivative of the function E(w) involves applying the chain rule and the properties of the logistic function. Let's denote the logistic function as θ(z) = (1 + e-z)-1. The derivative of E(w) with respect to w is ∇E(w) = ∂E/∂w. Using the chain rule:

∇E(w) = ΣNn=1(-ynxn(1 - θ(ynwTxn)))

This takes into account the derivative of the logarithm and the logistic function.

(b) Even though θ(wTx) is a nonlinear function, the decision boundary of logistic regression remains linear because it is determined by the set of points for which wTx = 0. This can be visualized as a line (or hyperplane in higher dimensions) in the input space. The nonlinearity of θ does not affect this boundary because it only scales the output between 0 and 1.

(c) The decision boundary is still linear if the prediction rule changes to use a threshold of 0.9 instead of 0.5. This is because the decision boundary is still determined by wTx being equal to the log-odds corresponding to θ-1(0.9), which is a constant. Essentially, changing the threshold shifts the boundary in input space but does not change its linearity.

(d) The essential property of logistic regression that results in a linear decision boundary is that the classification decision is made based on whether the linear combination of input features wTx crosses a threshold. Regardless of the value of this threshold, the boundary is defined by a set of points with equal linear score, which forms a hyperplane when graphed in the input feature space.