Statistics · Model fitting · Medium
The same regressor twice
A column x appears twice in a regression, as x₁ and x₂, with no intercept, and the least-squares slope of y on x alone is positive. You fit y on both copies with a lasso penalty: you minimise the sum of squared residuals plus λ × (|b₁| + |b₂|), where λ is positive but small enough that the best fit is not zero. Does this minimisation have exactly one solution, exactly two solutions or infinitely many solutions?
Answer and solution
Answer
Infinitely many solutions
Solution
With identical columns both the fit and the lasso penalty depend only on b₁ + b₂, so the penalty cannot choose how to split it.
Let s = b₁ + b₂ be the best total, which is positive. Any split with b₁ ≥ 0, b₂ ≥ 0 and b₁ + b₂ = s gives the same residuals and the same penalty λs, while a split with opposite signs has a larger penalty. Every point on that segment is a minimum: infinitely many solutions. A ridge penalty, λ(b₁² + b₂²), is smallest at the even split, so ridge has exactly one solution with b₁ = b₂.
Model fitting comes up in 6 firms' roundsTwo Sigma · Tower Research · Millennium · +3
These 8 rounds ask this kind of problem live, with follow-ups. Every round on every track is in Premium.