Orthogonal Complements

In this post, I want to talk about something called an orthogonal complement. The set B is the orthogonal complement of the set A if for all a\in A and for all b\in B, a\cdot b = 0. We write this as A^{\perp} = B since everything in A is orthogonal to everything in B.

Note that the definition of orthogonal complements is symmetric, meaning that if A^{\perp} = B, then we also know that B^{\perp} = A. This is because the dot product is commutative (a\cdot b=b\cdot a).

Let’s look at an example of a set and its orthogonal complement. Consider the xy-plane through the origin in \mathbb{R}^3 This can be written as span\left\{\begin{pmatrix}1\\0\\0\end{pmatrix},\begin{pmatrix}0\\1\\0\end{pmatrix}\right\}, or any vector that has 0 as its z component. So, the orthogonal complement of the xy-plane is any vector of the form \begin{pmatrix}0\\0\\k\end{pmatrix}, where k\in\mathbb{R}.

Now let’s look at an example of orthogonal complements in the context of matrix multiplication. To do this, we’ll relate two features of a matrix, the null space and the row space.

Consider an m\times n matrix A, which represents a transformation from \mathbb{R}^n to \mathbb{R}^m. The null space of A, written Nul(A) is the set of all x such that Ax = \vec{0}. The row space of A is the set of all linear combinations of the rows of A.

To understand why these two sets are orthogonal to each other, let’s write out matrix multiplication in a way that makes it easier to see. We’ll do this with an example of 3\times 3 matrix, but the idea extends to a matrix with any dimensions.

\begin{pmatrix} 1&2&2\\3&-1&-2\\5&-4&-3\end{pmatrix}\begin{pmatrix}x_1\\x_2\\x_3\end{pmatrix}.

We would normally understand this matrix vector multiplication as a weighted sum of the columns, or x_1\begin{pmatrix}1\\3\\5\end{pmatrix} + x_2\begin{pmatrix}2\\-1\\-4\end{pmatrix} +x_3\begin{pmatrix}2\\-2\\-3\end{pmatrix}, but we are going to look at it a different way.

Then, we can rewrite \begin{pmatrix} 1&2&2\\3&-1&-2\\5&-4&-3\end{pmatrix}\begin{pmatrix}x_1\\x_2\\x_3\end{pmatrix} as \begin{pmatrix}(1,2,2)\cdot (x_1,x_2,x_3)\\(3,-1,2)\cdot (x_1,x_2,x_3)\\(5,-4,-3)\cdot (x_1,x_2,x_3)\end{pmatrix}.

We can see that these two ways of multiplying matrices are the same because, for example, the first component of the final vector we just output is x_1+2x_2+2x_3. And if you multiply the x‘s through in the first definition of matrix multiplication above, you’ll get the same thing for the first component of the vector.

Now, let’s consider the vector \begin{pmatrix}1\\-2\\1\end{pmatrix}\in Nul\left\{\begin{pmatrix}1&2&3\\4&5&6\\7&8&9\end{pmatrix}\right\}. Then, \begin{pmatrix}1&2&3\\4&5&6\\7&8&9\end{pmatrix}  \begin{pmatrix}1\\-2\\1\end{pmatrix}=\begin{pmatrix}0\\0\\0\end{pmatrix}. Since \begin{pmatrix}1\\-2\\1\end{pmatrix} is in the null space of \begin{pmatrix}1&2&3\\4&5&6\\7&8&9\end{pmatrix}, it must be orthogonal to each row that matrix.

Now, all we need to show is that if a vector is orthogonal to each row, then it is orthogonal to any linear combination of those rows.

Suppose \vec{x} is orthogonal to every vector in the set \{\vec{a_1}, \cdots,\vec{a_m}\}. Then, let’s compute \vec{x}\cdot (c_1\vec{a_1}+\ldots \cdot c_m\vec{a_m}) (the right side of the dot is any vector in the span of the a_is). We can rewrite this as c_1(\vec{x}\cdot\vec{a_1})+\ldots + c_m(\vec{x}\cdot\vec{a_m}). We know that \vec{x}\cdot a_i=0 for 1\leq i\leq m, so this dot product is zero.

Thus, \vec{x} is orthogonal to any vector in the row space, and not just each vector that spans the row space.

Overall, we showed that if \vec{x}\in Nul(A), then \vec{x}\perp Row(A).

In the next post, we’ll connect this back to the Rank-Nullity theorem by talking about how the row space and column space of a matrix relate to each other.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s