Question

Consider stacking two logistic regression units on top of the input Rd for classification. In other words, we have

h = o(w x + b)
o (w2h+b2) y = o
where w, b, w2, b2 are parameters. Show that such a classifier cannot achieve non-linearity

Answer 1

Stacking two logistic regression units does not result in non-linearity since both units perform linear transformations followed by the logistic function, which maintains the linearity of the decision boundary.

Step-by-step explanation:

The student's question revolves around the properties of logistic regression and the possibility of stacking two logistic regression units for the purpose of classification.

To clarify, logistic regression by itself is a linear classifier, meaning that it can only create a linear decision boundary between classes. The logistic function, often denoted as o, is used to map predictions to probabilities and is defined by the logistic curve.

When stacking two logistic regression units with the structure h = o(w x + b)o (w2h+b2), each operation within this setup represents a linear transformation followed by a logistic function application.

Despite stacking two units, the operation remains within the realm of linear transformations, because the composition of a linear function with the logistic function (a sigmoid), which is applied element-wise and does not change the linearity of the mapping, still results in a linear decision boundary.

Therefore, the classifier cannot achieve non-linearity due to the inherited linear properties of logistic regression.