1

Given a vector $\vec{y}$, a nzonsquare matrix $M$, and a vector $\vec{\alpha}$, is it possible to solve for the diagonal square matrix $W$ which satisfies

$(M^TWM)^{-1}M^TW\vec{y} = \vec{\alpha}$?

Practically this arose as a statistical problem, where I have raw data and WLS fit observations, but not the design matrix or weights that link those things. I am able to solve for the design matrix on OLS data just using the pseudoinverse of the pseudo inverse, but I don't see a clear equivalent manipulation here. In particular that requires me to have multiple observation/fit data pairs all of which use the same matrix (to avoid singularity issues).

In this case, there is a different W being used for each pair, and I don't see a linear algebra solution. Trying to expand numerically results in a nonlinear system of equations defining W, which seems fairly hopeless.

librus
  • 147
  • 1
    Is it possible that there is a typo fro the classical WLS model $(M^TW^2M)^{-1}M^TW\vec{y}=\vec{\alpha}$? Or is there some implicit information on $W$ that yields the equation in your post? – cjferes Nov 30 '24 at 01:22
  • That is only equivalent if M is square. I have nonsquare M. Ignoring whether it should be W or W^2, my equation is the same on that appears on wikipedia for instance: https://en.wikipedia.org/wiki/Weighted_least_squares – librus Nov 30 '24 at 03:21

1 Answers1

2

I don't think there will be a unique solution for $W$ in general. Assume $p<n$.

The first-order condition for WLS says that $$M^\top Wr = 0,$$ where $r = y-\hat{y}$ is a known fixed vector.

This is a linear system in $w_1,\dots,w_n$ with $p$ equations and $n$ unknowns.

Moreover, any $\tilde W$ which satisfies this condition and for which $M^\top \tilde W M$ is invertible will lead to the same $\vec{\alpha}$:

$$(M^T\tilde WM)^{-1}M^T\tilde W\vec{y}=(M^T\tilde WM)^{-1}M^T\tilde WM\vec{\alpha} = \vec{\alpha}.$$

(This second condition will be true if all the weights are positive and $M$ has full column rank.)

We can find weights that work by solving the following system:

$$ \begin{align*} M^\top Wr &= 0,\\ \sum w_i &= 1,\\ w_i &\geq 0. \end{align*} $$

Here is a Pythonic simulation using and packages:

import numpy as np
import cvxpy as cp

n = 10
p = 3


X = np.random.randn(n, p)
w_truth = np.random.uniform(0.0, 1.0, size=n)
y = np.random.randn(n)


beta_WLS = np.linalg.inv(X.T @ np.diag(w_truth) @ X) @ (X.T @ np.diag(w_truth) @ y)
y_hat = X @ beta_WLS
r = y - y_hat


w = cp.Variable(n, nonneg=True)


constraints = [
    X.T @ cp.diag(w) @ r == 0,
    cp.sum(w) == 1.0
]


objective = cp.Minimize(0)


problem = cp.Problem(objective, constraints)
problem.solve()

The weights found are different from the ground truth w_truth:

> w.value

array([0.14236722, 0.11848218, 0.08266685, 0.07218026, 0.10800815,
       0.12393363, 0.08405078, 0.10545762, 0.08428544, 0.07856788])

> w_truth/np.sum(w_truth)

array([0.16970982, 0.09158631, 0.11019793, 0.12181479, 0.07958308,
       0.10987568, 0.10230436, 0.15777878, 0.00795696, 0.04919229])

but they both lead to the same $\vec{\alpha}$:

> np.linalg.inv(X.T @ np.diag(w.value) @ X) @ (X.T @ np.diag(w.value) @ y)

array([-0.33274219,  1.22240004, -0.29890388])

> beta_WLS

array([-0.33274219,  1.22240004, -0.29890388])
Mario
  • 37
DDD
  • 159