10

Suppose I have a linear subspace of some vector space, e.g., described as the column space of some big matrix. How would I algorithmically find a basis of that same subspace where the basis matrix is sparse, i.e., most entries in most basis vectors are zero?

I understand that this will depend on the structure of the matrix, so you might prefer to interpret this as “as many entries as possible are zero” instead of “most entries are zero”. I'm interested in good practical solutions, even if the results are not optimal.

Rodrigo de Azevedo
  • 23,223
MvG
  • 44,006
  • 2
    In general it can't be done. If your vector space is $d$-dimensional, generically you can find a nonzero vector where any set of $d-1$ entries are $0$. – Robert Israel Feb 26 '13 at 21:24
  • @RobertIsrael: clarified my question: I want as many zeros as possible, even if that might turn out to be less than I could wish for. Although I must confess I don't fully understand your comment. Is $d$ the dimension of the subspace or the enclosing space? And why do you state that many zeros are impossible and then describe a vector which has many zeros? – MvG Feb 26 '13 at 21:32
  • A basis may always be column operated on - think Gaussian reduction. So if somehow you are attempting to do better than some such reduction (as it seems since you mention the desire for many zeros) this is a difficult problem in general. What you are asking may actually be related to lattice reduction, which has no easy solution. – adam W Feb 26 '13 at 21:53
  • $d$ is the dimension of the subspace. If that's much less than the dimension of the enclosing space, a vector with $d-1$ zeros is not very sparse. – Robert Israel Feb 26 '13 at 22:03
  • @MvG Do you care about the conditioning of your basis? – Dominique Feb 26 '13 at 22:06
  • @Dominique: as I'll be doing exact rational arithmetic, conditioning doesn't matter. – MvG Feb 27 '13 at 06:24
  • 2
    related question: http://mathoverflow.net/questions/253995/how-can-one-construct-a-sparse-null-space-basis-using-recursive-lu-decomposition – cladelpino Jan 22 '17 at 06:41

2 Answers2

4

Suppose your subspace $V$ is the column space of the $m \times n$ matrix $M$, where $M$ has rank $n < m$. Choose any subset $S$ of $\{1,\ldots, m\}$ of cardinality $n-1$, and consider the $(n-1) \times n$ submatrix $M_S$ of $M$ consisting of the rows enumerated in $S$. If $y$ is a nonzero vector in the null space of $M_S$, then $M y$ is a nonzero member of $V$ with zeros in the positions given by $S$. Do this for $m$ randomly chosen subsets $S$ and and you will probably get enough such vectors to span $V$.

For example, I tried the random $7 \times 4$ matrix $$ M = \left[ \begin {array}{cccc} -6&2&3&9\\ 0&5&-1&2 \\ 1&-2&-1&7\\ 8&1&3&-2 \\ 7&-4&7&-8\\ -1&-1&-6&9 \\ -6&-4&-6&-6\end {array} \right] $$

Using subsets $[2, 3, 7], [4, 5, 7], [3, 5, 6], [3, 5, 7]$ I got the vectors consisting of the columns of $$\left[ \begin {array}{cccc} {\frac {780}{19}}&{\frac {3}{26}}&-{ \frac {289}{28}}&{\frac {681}{13}}\\ 0&-{\frac {80}{ 13}}&{\frac {471}{14}}&-{\frac {311}{26}}\\ 0&{ \frac {151}{13}}&0&0\\ -{\frac {468}{19}}&0&{\frac { 615}{14}}&-{\frac {755}{26}}\\ -{\frac {311}{19}}&0&0 &0\\ -4&{\frac {170}{13}}&0&-{\frac {415}{26}} \\ 0&0&-{\frac {415}{7}}&0\end {array} \right] $$

Since that matrix has rank $4$, these columns span $V$.

Robert Israel
  • 470,583
0

I suspect this is a tough problem to solve in one shot but here is an idea. Bear with me as I'm thinking out loud. Suppose you have already identified $a_1, \ldots, a_p \in V$ that are linearly independent. Construct $$ A := \begin{bmatrix} a_1 \cdots a_p \end{bmatrix}. $$ Suppose further that $V$ can be described by the linear system $Bx=d$. One way to find a sparse vector in $V$ that is orthogonal to $a_1, \ldots, a_p$ is to solve the $\ell_1$ problem $$ \min_x \ \|x\|_1 \quad \text{subject to} \ Bx=d, \ A^T x = 0. $$ This can be transformed to a linear programming problem as follows: \begin{align*} \min_{x,v} \quad & \sum_i v_i \\ \text{subject to} \quad & Bx = d, \\ & A^T x = 0, \\ & -v \leq x \leq v. \end{align*} This isn't guaranteed to find the sparsest solution but is generally speaking relatively successful at that. The idea is that the $\ell_1$ norm is the closest convex norm to the so-called $\ell_0$ norm, which isn't really a norm but simply counts the number on nonzeros of a vector.

Dominique
  • 3,224
  • Orthogonality isn't a requirement for me, but perhaps it might make zeros more likely. What I'm worried about is the fact that you're building the basis incrementally and greedily, so a choice early on might have adverse affects on later vectors, even if it seemed a hood choice at that time. I'm also not completely clear about the $Bx=d$ description. I'd read this as describing $V$ as the kernel of $B$, in which case $d$ would be zero and you'd need some extra constraint to ensure that $x$ is not zero. Right? Using $\ell_1$ sounds related to http://math.stackexchange.com/a/170410/35416. – MvG Feb 27 '13 at 06:32
  • @MvG: You're right. This is just an initial idea and it has flaws. Orthogonality seemed to be a simple way of imposing linear independence. And you're also right that if $d=0$, special care must be taken to avoid choosing $x=0$. – Dominique Feb 28 '13 at 03:01