02-LinearAlgebra.Rmd

# Linear Algebra


When formalizing intuitive mathematical ideas, we define a set of *objects* and *rules* for manipulating them—this structure is called an *algebra*. In particular, **linear algebra** focuses on **vectors** and the rules that govern how they can be **added** and **scaled**.  



<p align="center">
<img src="Figure2.1MML.png" alt="A mind map of the concepts introduced in this chapter, along with where they are used in other parts of the book." width="400">
</p>




---


<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->




## **2.0** Vectors {-}



<div class="definition">
A **vector** is a mathematical object that can be **added** to other vectors and **multiplied by scalars**, resulting in another vector of the same kind.  A vector $\mathbf{v} \in  \mathbb{R}^n$ has the form    \[
  \mathbf{v} =
  \begin{bmatrix}
   v_1 \\ v_2 \\ \vdots \\ v_n
   \end{bmatrix},
   \]
where each $v_i \in \mathbb{R}$.   
</div>



### Vector Spaces

While most people are familiar with *geometric vectors* (arrows with direction and magnitude), vectors can also take more abstract forms—as long as they obey the two key operations:

1. **Addition:** \( \mathbf{a} + \mathbf{b} = \mathbf{c} \)
2. **Scalar multiplication:** \( \lambda \mathbf{a} = \mathbf{b} \)


<div class="definition">
A set \( V \) is a **vector space** over \( \mathbb{R} \) if for any \( \mathbf{u}, \mathbf{v} \in V \) and any scalar \( \lambda \in \mathbb{R} \):

\[
\mathbf{u} + \mathbf{v} \in V \quad \text{and} \quad \lambda \mathbf{u} \in V
\]
</div>


<div class="definition">
Let \( \mathbf{u}, \mathbf{v} \in \mathbb{R}^n \) be two vectors.  The *sum* of two vectors is obtained by adding their corresponding components:  
   \[
   \mathbf{u} + \mathbf{v} =
   \begin{bmatrix}
   u_1 \\ u_2 \\ \vdots \\ u_n
   \end{bmatrix}
   +
   \begin{bmatrix}
   v_1 \\ v_2 \\ \vdots \\ v_n
   \end{bmatrix}
   =
   \begin{bmatrix}
   u_1 + v_1 \\ u_2 + v_2 \\ \vdots \\ u_n + v_n
   \end{bmatrix}
   \]
</div>




<div class="example">
We can add two vectors componentwise: 
\[
\begin{bmatrix} 1 \\ 4 \\ 10\\ 20 \end{bmatrix} + \begin{bmatrix} 2 \\ 6 \\ 15\\ 30 \end{bmatrix} = \begin{bmatrix} 1+2 \\ 4+6 \\ 10+15\\ 20+30 \end{bmatrix} = \begin{bmatrix} 2 \\ 10 \\ 25\\ 50 \end{bmatrix}.
\]
</div>


<div class="definition">
Let \( \mathbf{u} \in \mathbb{R}^n \) be a vector, and let \( \lambda \in \mathbb{R} \) be a scalar.  The *product* of a scalar \( \lambda \) and a vector \( \mathbf{u} \) is obtained by multiplying each component of the vector by the scalar:  
   \[
   \lambda \mathbf{u} =
   \lambda
   \begin{bmatrix}
   u_1 \\ u_2 \\ \vdots \\ u_n
   \end{bmatrix}
   =
   \begin{bmatrix}
   \lambda u_1 \\ \lambda u_2 \\ \vdots \\ \lambda u_n
   \end{bmatrix}
   \]
</div>




<div class="example">
Scalar multiplication is applied to each term:
\[
5\begin{bmatrix} 1 \\ 4 \\ 10\\ 20 \end{bmatrix} = \begin{bmatrix} 5 \times 1 \\ 5 \times 4 \\ 5 \times 10\\ 5 \times 20 \end{bmatrix} = \begin{bmatrix} 5 \\ 20 \\ 50\\ 100 \end{bmatrix}.
\]
</div>

<div class="example">
The set of complex numbers $\mathbb{C}$ is a vector space.  To prove this, we need to show that it satisfies the two properties:

1.  For $u, v \in \mathbb{C}$, we have $u+v \in \mathbb{C}$.
2.  For $u \in \mathbb{C}$ and $\lambda \in \mathbb{R}$, we have $\lambda u \in \mathbb{C}$.

Let $u,v \in \mathbb{C}$.  Then $u = a+bi$ and $v = c + di$.

1.  **Vector addition:**  The usual complex addition is defined as:
$$
u + v = (a + bi) + (c + di) = (a + c) + (b + d)i \in \mathbb{C}
$$

2.  **Scalar multiplication:** For real scalars \( r \in \mathbb{R} \), scalar multiplication is defined as:
$$
r \cdot (a + bi) = (ra) + (rb)i \in \mathbb{C}.
$$
</div>


Other examples of Vector spaces include:

- **Geometric vectors:** Can be drawn in space and manipulated visually.  
- **Polynomials:** Can be added and scaled to form new polynomials.  
- **Audio signals:** Represented as sequences of numbers that can be combined or scaled.  
- **Tuples of real numbers** in \( \mathbb{R}^n \) 



Treating vectors as elements of \( \mathbb{R}^n \) aligns with how data is represented in computer programs—arrays of real numbers.   This makes linear algebra essential for computational work and for algorithms in machine learning and data science.




---


###  Closure

A central idea in mathematics, known as **closure**, asks what new elements can be formed by combining existing ones through defined operations.  In linear algebra, the set of all possible linear combinations of vectors forms a **vector space**, a foundational concept throughout machine learning.


<div class="definition">
The **closure property** (or simply *closure*) describes whether a set is **closed under an operation** — meaning that when the operation is applied to elements of the set, the result is also an element of the same set.  

Formally, a set \( S \) is **closed** under an operation \( \circ \) if for all \( a, b \in S \):
\[
a \circ b \in S
\]
</div>



<div class="example">
The set of real numbers \( \mathbb{R} \) is **closed under addition** because for any \( a, b \in \mathbb{R} \), the sum \( a + b \in \mathbb{R} \).
</div>

<div class="example">
The set of integers \( \mathbb{Z} \) is **not closed under division**, since \( 1 \div 2 = 0.5 \notin \mathbb{Z} \).
</div>

<div class="example">
The set of vectors of a set length, $n$, is closed since $\mathbf{a}, \mathbf{b} \in \mathbb{R}^n$ implies that \[\mathbf{a} + \mathbf{b} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} + \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix} = \begin{bmatrix} a_1+b_1 \\ a_2+b_2 \\ \vdots \\ a_n+b_n \end{bmatrix}.
\]

However, the set of all vectors is not closed since $\begin{bmatrix} 1 \\ 2 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \\3 \end{bmatrix}$ is undefined.
</div>




---


###  Other Properties of Vectors


Vectors obey a set of algebraic rules that make them fundamental in both geometry and linear algebra.  

<div class="lemma">
Let \( \mathbf{u}, \mathbf{v}, \mathbf{w} \) be vectors, and let \( c, d \) be scalars.  

1. **Commutativity of Addition**  
   \[
   \mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}
   \]

2. **Associativity of Addition**  
   \[
   (\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})
   \]

3. **Additive Identity**  
   There exists a **zero vector** \( \mathbf{0} \) such that  
   \[
   \mathbf{v} + \mathbf{0} = \mathbf{v}
   \]

4. **Additive Inverse**  
   For every vector \( \mathbf{v} \), there exists a vector \( -\mathbf{v} \) such that  
   \[
   \mathbf{v} + (-\mathbf{v}) = \mathbf{0}
   \]

5. **Distributive Properties**  
   \[
   c(\mathbf{u} + \mathbf{v}) = c\mathbf{u} + c\mathbf{v}
   \]
   \[
   (c + d)\mathbf{v} = c\mathbf{v} + d\mathbf{v}
   \]

6. **Associativity of Scalar Multiplication**  
   \[
   c(d\mathbf{v}) = (cd)\mathbf{v}
   \]

7. **Multiplicative Identity**  
   \[
   1 \mathbf{v} = \mathbf{v}
   \]
</div>


Sometimes, we use row vectors rather than column vectors simply to save space or for aesthetic reasons.

<div class="example">
To prove the commutative rule for vector addition, write \(\mathbf{u}=[u_1,\dots,u_n]\), \(\mathbf{v}=[v_1,\dots,v_n]\). Addition is componentwise:
\[
u + v = [u_1+v_1,\dots,u_n+v_n].
\]
By commutativity in \(\mathbb{R}\),
\[
u_i + v_i = v_i + u_i \quad \forall i,
\]
so
\[
\mathbf{u} + \mathbf{v} = [v_1+u_1, \dots, v_n+u_n] = \mathbf{v} + \mathbf{u}.
\]
</div>


The **dot product** (also called the **inner product**) is an operation that takes two vectors and returns a single number. It measures how similar or aligned the two vectors are.


<div class="definition">
For vectors \( \mathbf{a}, \mathbf{b} \in \mathbb{R}^n \), the dot product is defined as:
\[
\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i
\]
</div>

<div class="example">
Compute the dot product of $\begin{bmatrix} 2 \\ 5 \\ 4 \end{bmatrix}$ and $\begin{bmatrix} -3\\ 0 \\ -2 \end{bmatrix}$.
\begin{align*}
\begin{bmatrix} 2 \\ 5 \\ 4 \end{bmatrix} \cdot \begin{bmatrix} -3\\ 0 \\ -2 \end{bmatrix} &= \sum_{i=1}^{3} a_i b_i\\
&= a_1b_1 + a_2 b_2 + a_3 b_3\\
&= 2(-3) + 5(0) + 4(-2)\\
&= -14
\end{align*}
</div>






---


### Geometric Interpretation of a Vector




Geometrically, vectors can be thought of as **arrows** that have both **magnitude** (length) and **direction**. They are often used to represent quantities such as displacement, velocity, or force.  

Vector addition corresponds to placing one arrow’s tail at the head (the *triangle rule*), resulting in a new vector that represents the combined effect of both. 


<p align="center">
<img src="TipToTail2.png" alt="https://courses.lumenlearning.com/ccbcmd-math/chapter/performing-vector-addition-and-scalar-multiplication/" width="400">
</p>

Scalar multiplication stretches or shrinks a vector and can reverse its direction if the scalar is negative.  


<p align="center">
<img src="ScalarMultiplication2.png" alt="https://courses.lumenlearning.com/ccbcmd-math/chapter/performing-vector-addition-and-scalar-multiplication/" width="400">
</p>


These geometric operations follow the same algebraic properties found in vector spaces—such as commutativity, associativity, and distributivity — allowing us to interpret abstract vector operations visually as movements and scalings in space.  





<div class="example">
Let $\mathbf{u} = [3,-2]$ and $\mathbf{v} = [-1,4]$.  Then $\mathbf{u} + \mathbf{v}$ and $\mathbf{u} - \mathbf{v}$ can be computed using vectors:
\[\mathbf{u} + \mathbf{v}= [2,2] \;\;\; \text{ and } \;\;\; \mathbf{u} - \mathbf{v} = [4,-6].\]

<p align="center">
<img src="VectorAdditionExample.png" alt="https://courses.lumenlearning.com/ccbcmd-math/chapter/performing-vector-addition-and-scalar-multiplication/" width="400">
</p>
</div>




---


### Exercises {.unnumbered .unlisted}


<div class="exercise">
Vector addition

<div style="text-align: right;">
[Solution]( )
</div> 

</div><div class="exercise">
Scalar Multiplication

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

</div><div class="exercise">
Both

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Show that each of the following are vector spaces:

-  $\mathbb{R}^n$  
-  Polynomials  
-  Continuous functions  
-  Sequences  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
Let \( \mathbf{u}, \mathbf{v}, \mathbf{w} \) be vectors, and let \( c, d \) be scalars. Prove each of the following properties: 

1. **Commutativity of Addition**  
   \[
   \mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}
   \]

2. **Associativity of Addition**  
   \[
   (\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})
   \]

3. **Additive Identity**  
   There exists a **zero vector** \( \mathbf{0} \) such that  
   \[
   \mathbf{v} + \mathbf{0} = \mathbf{v}
   \]

4. **Additive Inverse**  
   For every vector \( \mathbf{v} \), there exists a vector \( -\mathbf{v} \) such that  
   \[
   \mathbf{v} + (-\mathbf{v}) = \mathbf{0}
   \]

5. **Distributive Properties**  
   \[
   c(\mathbf{u} + \mathbf{v}) = c\mathbf{u} + c\mathbf{v}
   \]
   \[
   (c + d)\mathbf{v} = c\mathbf{v} + d\mathbf{v}
   \]

6. **Associativity of Scalar Multiplication**  
   \[
   c(d\mathbf{v}) = (cd)\mathbf{v}
   \]

7. **Multiplicative Identity**  
   \[
   1 \mathbf{v} = \mathbf{v}
   \]
<div style="text-align: right;">
[Solution]( )
</div> 
</div>



<div class="exercise">
Dot product example


<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">

<div style="text-align: right;">
[Solution]( )
</div> 
</div>



---










<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->








## Systems of Linear Equations



Systems of linear equations are fundamental in linear algebra, as many problems can be formulated in this way. Linear algebra provides the tools to solve these systems efficiently.


<div class="example">
A company produces products \( N_1, \ldots, N_n \) using resources \( R_1, \ldots, R_m \).  Each product \( N_j \) requires \( a_{ij} \) units of resource \( R_i \).  If \( b_i \) units of each resource \( R_i \) are available, then the total resources used must satisfy

\[
a_{i1}x_1 + a_{i2}x_2 + \cdots + a_{in}x_n = b_i, \quad i = 1, \ldots, m
\]

or, in general matrix form,

\[
A\mathbf{x} = \mathbf{b},
\]

where \( A = [a_{ij}] \in \mathbb{R}^{m \times n} \), \( \mathbf{x} \in \mathbb{R}^n \), and \( \mathbf{b} \in \mathbb{R}^m \).
</div>


<div class="example">
Suppose a factory produces two products: **P1** and **P2**.  
Let:

- \(x_1\) = units of P1 produced  
- \(x_2\) = units of P2 produced  

**Constraints:**

1. Labor: Each unit of P1 requires 2 hours, P2 requires 3 hours, and total available labor is 120 hours:  
   \[
   2x_1 + 3x_2 \leq 120
   \]  
2. Material: Each unit of P1 uses 1 kg of material, P2 uses 2 kg, and total available material is 100 kg:  
   \[
   1x_1 + 2x_2 \leq 100
   \]  
3. Non-negativity:  
   \[
   x_1 \geq 0, \quad x_2 \geq 0
   \]

We can rewrite the inequalities as a matrix inequality:
\[
\underbrace{
\begin{pmatrix}
2 & 3 \\
1 & 2
\end{pmatrix}}_{A}
\begin{pmatrix}
x_1 \\ x_2
\end{pmatrix}
\leq
\underbrace{
\begin{pmatrix}
120 \\ 100
\end{pmatrix}}_{b}
\]

Here:

- \(\mathbf{A}\) is the **constraint matrix**.  
- \(x = \begin{pmatrix}x_1 \\ x_2\end{pmatrix}\) is the **decision variable vector**.  
- \(b\) is the **resource vector**.  
</div>


---


###  Solutions to Systems of Linear Equations


In general, a system of linear equations can have:

- no solution,  
- exactly one solution, or  
- infinitely many solutions.  

<div class="example">
The system
\[
\begin{aligned}
x_1 + x_2 + x_3 &= 3 \\
x_1 - x_2 + 2x_3 &= 2 \\
2x_1 + 3x_3 &= 1
\end{aligned}
\]
has **no solution**, since combining the first two equations gives \( 2x_1 + 3x_3 = 5 \), which contradicts the third equation.  
</div>

Systems of equations with no solutions are called *inconsistent*.

<div class="definition">
A system of linear equations is said to be **inconsistent** if no set of values for the unknown variables satisfies all equations simultaneously.   
</div>

In other words, the equations contradict each other, and there is no common solution. 


<div class="example">
The system
\[
\begin{aligned}
x_1 + x_2 + x_3 &= 3 \\
x_1 - x_2 + 2x_3 &= 2 \\
x_2 + x_3 &= 2
\end{aligned}
\]
has **a unique solution** \( (x_1, x_2, x_3) = (1, 1, 1) \).
</div>


<div class="example">
The system
\[
\begin{aligned}
x_1 + x_2 + x_3 &= 3 \\
x_1 - x_2 + 2x_3 &= 2 \\
2x_1 + 3x_3 &= 5
\end{aligned}
\]
has **infinitely many solutions**, since the third equation is a linear combination of the first two.  
If we set \( x_3 = a \in \mathbb{R} \), then
\[
(x_1, x_2, x_3) =
\left( \tfrac{5}{2} - \tfrac{3}{2}a,\,
\tfrac{1}{2} + \tfrac{1}{2}a,\,
a \right),
\quad a \in \mathbb{R}.
\]
Hence, the solution set forms a line in \(\mathbb{R}^3\).
</div>




<div class="definition">
A system of linear equations is called **consistent** if there exists at least one set of values for the unknown variables that satisfies all equations simultaneously.  

- If there is exactly one solution, the system is **uniquely consistent**.  
- If there are infinitely many solutions, the system is **dependent** (still consistent).  
</div>


<div class="example">
The system of equations\[
\begin{cases}
x + y = 5 \\
2x - y = 1
\end{cases}
\]
is a consistent system with a unique solution: \(x = 2\), \(y = 3\).
</div>


<div class="example">
The system of equations 
\[
\begin{cases}
x + y = 4 \\
2x + 2y = 10
\end{cases}
\]
is inconsistent since the second equation is equivalent to \(x + y = 5\), which contradicts the first equation (\(x + y = 4\)).  Thus, the system has no solution.
</div>


---

### Geometric Interpretation

In two dimensions, each linear equation represents a **line** on the \( x_1x_2 \)-plane. The **solution set** is the intersection of these lines, which can be:

- a **point** (unique solution),  
- a **line** (infinitely many solutions), or  
- **empty** (no solution).  

For three variables, each equation defines a **plane** in \( \mathbb{R}^3 \).  Their intersection can be a plane, line, point, or empty set.


<div class="example">
\[
\begin{aligned}
4x_1 + 4x_2 &= 5 \\
2x_1 - 4x_2 &= 1
\end{aligned}
\]
has the unique solution \( (x_1, x_2) = (1, \tfrac{1}{4}) \).
</div>


<div class="example">
Consider the system:
\[
\begin{cases}
x + y + z = 3 \\
2x + 2y + 2z = 6 \\
x - y + z = 1
\end{cases}
\]
The second equation is just \(2 \times\) the first equation, so it doesn't add a new constraint.  The first and third equations define a plane intersection.  This leaves one free variable, so there are infinitely many solutions.


Let \(z = t\) (free parameter), then:
\[
\begin{aligned}
x + y + t &= 3 \implies x = 3 - y - t \\
x - y + t &= 1 \implies (3 - y - t) - y + t = 1 \implies 2y = 2 \implies y = 1 \\
x &= 3 - 1 - t = 2 - t
\end{aligned}
\]
Thus, the general solution is:
\[
(x, y, z) = (2 - t, 1, t), \quad t \in \mathbb{R}.
\]
</div>



---

### Matrix Formulation

A system of linear equations can be written compactly as:

\[
\begin{bmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix}
\begin{bmatrix}
x_1 \\ \vdots \\ x_n
\end{bmatrix}
=
\begin{bmatrix}
b_1 \\ \vdots \\ b_m
\end{bmatrix}
\]

This matrix representation \( \mathbf{A}\mathbf{x} = \mathbf{b} \) provides a compact and powerful way to describe and solve systems of linear equations.



<div class="example">
Write the system of equations as a matrix: \[
\begin{cases}
x + y = 5 \\
2x - y = 1
\end{cases}
\]

The matrix version of this system is:
\[
\begin{bmatrix}
1 & 1 \\
2 & -1
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix}
=
\begin{bmatrix}
5 \\ 1
\end{bmatrix}
\]
</div>


<div class="example">
Write the system of equations as a matrix: \[\begin{cases}
x + y = 4 \\
2x + 2y = 10
\end{cases}
\]

The matrix version of this system is:
\[
\begin{bmatrix}
1 & 1 \\
2 & 2
\end{bmatrix}
\begin{bmatrix}
x \\ y
\end{bmatrix}
=
\begin{bmatrix}
4 \\ 10
\end{bmatrix}
\]
</div>



---


### Exercises {.unnumbered .unlisted}



<div class="exercise">
Write a system of equations with a unique solution.

</div>


<div class="exercise">
Write a system of equations with infinitely many solutions. 
</div>


<div class="exercise">
Write a system of equations with no solutions. 
</div>



<div class="exercise">
Solve example 2.11

<div style="text-align: right;">
[Solution]( )
</div> 
</div>



<div class="exercise">
Write 2.15 and 2.16 as a matrix equations

<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Solve a system of equations

<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Solve a system of equations

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Turn into a matrix form

<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Change from matrix to equations

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

---




<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->


## Matrices


Matrices play a central role in linear algebra. They provide a compact way to represent *systems of linear equations* and also serve as representations of linear functions (or mappings).  



<div class="definition">
A **matrix** is a rectangular array of numbers arranged in $m$ rows and $n$ columns.  
Formally, a real-valued matrix \( \mathbf{A} \in \mathbb{R}^{m \times n} \) is:

\[
\mathbf{A} = 
\begin{bmatrix}
a_{11} & a_{12} & \dots & a_{1n} \\
a_{21} & a_{22} & \dots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \dots & a_{mn}
\end{bmatrix}
\]
</div>

The notation $(\mathbf{A})_{ij}$ refers to the $ij^{th}$ element of $\mathbf{A}$.  So, $(\mathbf{A})_{ij} = a_{ij}$.  For example, if \[
\mathbf{A} =
\begin{bmatrix}
2 & 4 \\
1 & 3
\end{bmatrix},
\]
Then $\mathbf{A}_{11} = 2, \mathbf{A}_{12} = 4, \mathbf{A}_{21} = 1$ and $\mathbf{A}_{22} = 3$.

Matrices with one row are called **row vectors**, and those with one column are **column vectors**.

---

### Matrix Addition

<div class="definition">
For two matrices \( \mathbf{A}, \mathbf{B} \in \mathbb{R}^{m \times n} \), their sum and difference are defined element-wise:
\[
(\mathbf{A} + \mathbf{B})_{ij} = a_{ij} + b_{ij} \;\;\;\;\;\;\;\;\; (\mathbf{A} - \mathbf{B})_{ij} = a_{ij} - b_{ij}
\]
</div>

The result of matrix addition is another \( m \times n \) matrix.

<div class="example">
Let:
\[
\mathbf{A} =
\begin{bmatrix}
2 & 4 \\
1 & 3
\end{bmatrix},
\quad
\mathbf{B} =
\begin{bmatrix}
5 & 0 \\
-2 & 1
\end{bmatrix}.
\]
Then \[
\mathbf{A} + \mathbf{B} =
\begin{bmatrix}
2 + 5 & 4 + 0 \\
1 + (-2) & 3 + 1
\end{bmatrix}
=
\begin{bmatrix}
7 & 4 \\
-1 & 4
\end{bmatrix}
\]
\[
\mathbf{A} - \mathbf{B} =
\begin{bmatrix}
2 - 5 & 4 - 0 \\
1 - (-2) & 3 - 1
\end{bmatrix}
=
\begin{bmatrix}
-3 & 4 \\
3 & 2
\end{bmatrix}.
\]
</div>

<div class="example">
In the previous example, we can see that \[(\mathbf{A}+\mathbf{B})_{21} = -1 \quad \text{and} \quad (\mathbf{A}-\mathbf{B})_{22} = 2.\]
</div>

---

### Matrix Multiplication

<div class="definition">
For matrices \( \mathbf{A} \in \mathbb{R}^{m \times n} \) and \( \mathbf{B} \in \mathbb{R}^{n \times k} \), their product \( \mathbf{C} = \mathbf{A}\mathbf{B} \in \mathbb{R}^{m \times k} \) is defined as:
\[
c_{ij} = \sum_{l=1}^{n} a_{il} b_{lj}
\]
</div>

That is, each element of \( \mathbf{C} \) is obtained by taking the dot product of the corresponding row of \( \mathbf{A} \) and column of \( \mathbf{B} \).

<div class="note">
Matrix multiplication is only defined when the inner dimensions match (the number of columns of \( \mathbf{A} \) equals the number of rows of \( \mathbf{B} \)).  
</div> 


<div class="note">
Matrix multiplication is **not commutative**, meaning \( \mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A} \) in general.
</div> 


<div class="example">
Let
\[
\mathbf{A} =
\begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix},
\quad
\mathbf{B} =
\begin{bmatrix}
2 & 0 \\
1 & 2
\end{bmatrix}.
\]

Then
\[
\mathbf{A} \mathbf{B} =
\begin{bmatrix}
1(2) + 2(1) & 1(0) + 2(2) \\
3(2) + 4(1) & 3(0) + 4(2)
\end{bmatrix}
=
\begin{bmatrix}
4 & 4 \\
10 & 8
\end{bmatrix}.
\]
</div>


<div class="example">
Let
\[
\mathbf{C} =
\begin{bmatrix}
1 & 2 & 3 \\
4 & 5 & 6
\end{bmatrix},
\quad
\mathbf{D} =
\begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}.
\]

Then
\[
\mathbf{C} \mathbf{D} =
\begin{bmatrix}
1(1) + 2(3) + 3(5) & 1(2) + 2(4) + 3(6) \\
4(1) + 5(3) + 6(5) & 4(2) + 5(4) + 6(6)
\end{bmatrix}
=
\begin{bmatrix}
22 & 28 \\
49 & 64
\end{bmatrix}.
\]
</div>

---

### Identity Matrix


<div class="definition">
The **identity matrix** \( \mathbf{I}_n \in \mathbb{R}^{n \times n} \) is a square matrix with 1’s on the diagonal and 0’s elsewhere:
\[
\mathbf{I}_n =
\begin{bmatrix}
1 & 0 & \dots & 0 \\
0 & 1 & \dots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \dots & 1
\end{bmatrix}
\]
</div>

The identity matrix satisfies:
\[
\mathbf{I}_m \mathbf{A} = \mathbf{A} \mathbf{I}_n = \mathbf{A}
\] for any matrix $\mathbf{A}$ (with the appropriate dimensions).


<div class="example">
Let
\[
\mathbf{A} =
\begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix}
.\]
Then \[\mathbf{A}\mathbf{I} = \begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix}
\begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix}= 
\begin{bmatrix}
1(1) + 2(0) & 1(0) + 2(1) \\
3(1) + 4(0) & 3(0) + 4(1)
\end{bmatrix}= 
\begin{bmatrix}
1 & 2 \\
3 & 4
\end{bmatrix}.\]
</div>

---

### Matrix Properties

There are several important properties of matrices.



<div class="lemma">
For any matrices $\mathbf{A}, \mathbf{B}$, and $\mathbf{C}$ with appropriate dimensions for addition/ multiplication, the following properties hold.

- **Associativity:**  
  \[
  (\mathbf{A}\mathbf{B})\mathbf{C} = \mathbf{A}(\mathbf{B}\mathbf{C})
  \]  
- **Distributivity:**  
  \[
  (\mathbf{A} + \mathbf{B})\mathbf{C} = \mathbf{A}\mathbf{C} + \mathbf{B}\mathbf{C}, \quad \mathbf{A}(\mathbf{C} + \mathbf{D}) = \mathbf{A}\mathbf{C} + \mathbf{A}\mathbf{D}
  \]  
- **Identity Property:**  
  \[
  \mathbf{I}_m \mathbf{A} = \mathbf{A} \mathbf{I}_n = \mathbf{A}
  \]  
</div>


<div class="note"> 
Matrix multiplication is **not element-wise**. When multiplication is performed element by element, it is called the **Hadamard product**.
</div>


---




### Matrix Inverse




<div class="definition">
A square matrix \( \mathbf{A} \in \mathbb{R}^{n \times n} \) is **invertible** (or **nonsingular**) if there exists a matrix \( \mathbf{B} \in \mathbb{R}^{n \times n} \) such that  \[ \mathbf{A}\mathbf{B} = \mathbf{I}_n = \mathbf{B}\mathbf{A} .\]  
In this case, \( \mathbf{B} \) is called the **inverse** of \( \mathbf{A} \) and is denoted \( \mathbf{A}^{-1} \).  
</div>

If no such matrix exists, \( \mathbf{A} \) is **singular** or **noninvertible**. 

<div class="example">
Let
\[
\mathbf{A} =
\begin{bmatrix}
2 & 1 \\
7 & 4
\end{bmatrix},
\quad
\mathbf{B} =
\begin{bmatrix}
4 & -1 \\
-7 & 2
\end{bmatrix}.
\]
We will show that \( \mathbf{A} \) and \( \mathbf{B} \) are inverses of each other by verifying that:
\[
\mathbf{A} \mathbf{B} = \mathbf{B} \mathbf{A} = \mathbf{I}
\]
Notice that
\[
\mathbf{A} \mathbf{B} =
\begin{bmatrix}
2 & 1 \\
7 & 4
\end{bmatrix}
\begin{bmatrix}
4 & -1 \\
-7 & 2
\end{bmatrix}
=
\begin{bmatrix}
(2)(4) + (1)(-7) & (2)(-1) + (1)(2) \\
(7)(4) + (4)(-7) & (7)(-1) + (4)(2)
\end{bmatrix}
=
\begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix}.
\]
A similar calculation shows that $\mathbf{B}\mathbf{A} = \mathbf{I}$.
</div>


<div class="theorem">
The inverse of a matrix $\mathbf{A}$, when it exists, is **unique**.
</div>


For a 2×2 matrix  
\[
\mathbf{A} = 
\begin{bmatrix}
a_{11} & a_{12} \\
a_{21} & a_{22}
\end{bmatrix},
\]
the inverse is  
\[
\mathbf{A}^{-1} = \frac{1}{a_{11}a_{22} - a_{12}a_{21}}
\begin{bmatrix}
a_{22} & -a_{12} \\
-a_{21} & a_{11}
\end{bmatrix},
\]
provided \( a_{11}a_{22} - a_{12}a_{21} \neq 0 \).  The term \( a_{11}a_{22} - a_{12}a_{21} \) is the **determinant** of \( \mathbf{A} \).


<div class="example">
Let  
\[
\mathbf{A} =
\begin{bmatrix}
4 & 3 \\
2 & 1
\end{bmatrix}.
\]
The determinant is:
\[
\det(\mathbf{A}) = (4)(1) - (3)(2) = 4 - 6 = -2.
\]
Applying the formula gives us the inverse of $\mathbf{A}$:
\[
\mathbf{A}^{-1} = \frac{1}{-2}
\begin{bmatrix}
1 & -3 \\
-2 & 4
\end{bmatrix}
=
\begin{bmatrix}
-\tfrac{1}{2} & \tfrac{3}{2} \\
1 & -2
\end{bmatrix}.
\]
</div>

<div class="lemma">
For a square non-singular matrix $\mathbf{A}$, the following are true:

1.  $\mathbf{A}\mathbf{A}^{-1} = \mathbf{I} = \mathbf{A}^{-1}\mathbf{A}$,  
2.  $(\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}$,  
3.  $(\mathbf{A} + \mathbf{B})^{-1} \neq \mathbf{A}^{-1} + \mathbf{B}^{-1}$  
</div>




<div class="example">
Let
\[
\mathbf{A} =
\begin{bmatrix}
1 & 0 \\
0 & 2
\end{bmatrix},
\quad
\mathbf{B} =
\begin{bmatrix}
2 & 0 \\
0 & 1
\end{bmatrix}
\]
Then, using the definition of the 2x2 inverse,
\[
\mathbf{A}^{-1} =
\begin{bmatrix}
1 & 0 \\
0 & \tfrac{1}{2}
\end{bmatrix},
\quad
\mathbf{B}^{-1} =
\begin{bmatrix}
\tfrac{1}{2} & 0 \\
0 & 1
\end{bmatrix}
\]
So, their sum is
\[
\mathbf{A}^{-1} + \mathbf{B}^{-1} =
\begin{bmatrix}
1 + \tfrac{1}{2} & 0 \\
0 & \tfrac{1}{2} + 1
\end{bmatrix}
=
\begin{bmatrix}
\tfrac{3}{2} & 0 \\
0 & \tfrac{3}{2}
\end{bmatrix}
\]

On the other hand,
\[
\mathbf{A} + \mathbf{B} =
\begin{bmatrix}
1 + 2 & 0 \\
0 & 2 + 1
\end{bmatrix}
=
\begin{bmatrix}
3 & 0 \\
0 & 3
\end{bmatrix}.
\]
Using the definition of the inverse, we get:
\[
(\mathbf{A} + \mathbf{B})^{-1} =
\begin{bmatrix}
\tfrac{1}{3} & 0 \\
0 & \tfrac{1}{3}
\end{bmatrix}
\]
Comparing, we see
\[
(\mathbf{A} + \mathbf{B})^{-1} =
\begin{bmatrix}
\tfrac{1}{3} & 0 \\
0 & \tfrac{1}{3}
\end{bmatrix}
\quad \text{vs.} \quad
\mathbf{A}^{-1} + \mathbf{B}^{-1} =
\begin{bmatrix}
\tfrac{3}{2} & 0 \\
0 & \tfrac{3}{2}
\end{bmatrix}.
\]
Clearly,
\[
(\mathbf{A} + \mathbf{B})^{-1} \neq \mathbf{A}^{-1} + \mathbf{B}^{-1}.
\]
</div>

---

### Matrix Transpose

<div class="definition">
The **transpose** of \( \mathbf{A} \in \mathbb{R}^{m \times n} \) is \( \mathbf{A}^\top \in \mathbb{R}^{n \times m} \), obtained by interchanging rows and columns.  
</div>


<div class="example">
Let  
\[
\mathbf{A} =
\begin{bmatrix}
1 & 4 & 7 \\
2 & 5 & 8
\end{bmatrix}.
\]
Matrix \(\mathbf{A}\) is a \(2 \times 3\) matrix (2 rows, 3 columns).  The transpose of \(\mathbf{A}\), denoted \(\mathbf{A}^\top\), is formed by turning rows into columns:
\[
\mathbf{A}^\top =
\begin{bmatrix}
1 & 2 \\
4 & 5 \\
7 & 8
\end{bmatrix}.
\]
Now \(\mathbf{A}^\top\) is a \(3 \times 2\) matrix (3 rows, 2 columns).
</div>


<div class="lemma">
For $\mathbf{A} \in \mathbb{R}^{m \times n}$ and $\mathbf{B} \in \mathbb{R}^{n \times p}$, the following properties hold:

1.  $(\mathbf{A}^\top)^\top = \mathbf{A}$,  
2.  $(\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top \mathbf{A}^\top$,  
3.  $(\mathbf{A} + \mathbf{B})^\top = \mathbf{A}^\top + \mathbf{B}^\top$.
</div>


<div class="example">
Prove that for any two matrices \(\mathbf{A}\) and \(\mathbf{B}\) of the same size,  
\[
(\mathbf{A} + \mathbf{B})^\top = \mathbf{A}^\top + \mathbf{B}^\top.
\]

**Proof:**
Let \(\mathbf{A} = [a_{ij}]\) and \(\mathbf{B} = [b_{ij}]\) be \(m \times n\) matrices.  
Then the sum \(\mathbf{A} + \mathbf{B}\) is defined elementwise as:
\[
(\mathbf{A} + \mathbf{B})_{ij} = a_{ij} + b_{ij}.
\]
The transpose of a matrix swaps its rows and columns.  So, the \((i, j)\)-th entry of \((\mathbf{A} + \mathbf{B})^\top\) is:
\[
(\mathbf{A} + \mathbf{B})^\top_{ij} = (\mathbf{A} + \mathbf{B})_{ji} = a_{ji} + b_{ji}.
\]

Now, consider \(\mathbf{A}^\top + \mathbf{B}^\top\).  The \((i, j)\)-th entry of this matrix is:
\[
(\mathbf{A}^\top + \mathbf{B}^\top)_{ij} = \mathbf{A}^\top_{ij} + \mathbf{B}^\top_{ij} = a_{ji} + b_{ji}.
\]

Since the corresponding entries are equal for all \(i, j\),
\[
(\mathbf{A} + \mathbf{B})^\top = \mathbf{A}^\top + \mathbf{B}^\top.
\]
</div>


---

### Symmetric Matrices



<div class="definition"> 
\mathbf{A} matrix \( \mathbf{A} \in \mathbb{R}^{n \times n} \) is **symmetric** if \( \mathbf{A} = \mathbf{A}^\top \).  
</div>

Only square matrices can be symmetric.  If \( \mathbf{A} \) is invertible, then \( \mathbf{A}^\top \) is also invertible and  
\[
(\mathbf{A}^{-1})^\top = (\mathbf{A}^\top)^{-1}.
\]
The sum of symmetric matrices is symmetric, but their product generally is not.


<div class="example">
Matrix \(\mathbf{A} \in \mathbb{R}^{4 \times 4}\) is symmetric
\[
\mathbf{A} =
\begin{pmatrix}
2 & 1 & 0 & -1 \\
1 & 3 & 4 & 2 \\
0 & 4 & 5 & 3 \\
-1 & 2 & 3 & 6
\end{pmatrix}.
\]
Notice that since $\mathbf{A}$ is symmetric, we have $(\mathbf{A})_{ij} = (\mathbf{A})_{ji}$.
</div>

---



###  Scalar Multiplication


Matrices scale the same way that vectors do.

<div class="definition">
For \( \mathbf{A} \in \mathbb{R}^{m \times n} \), and $\lambda \in \mathbb{R}$, **scalar multiplication** is defined componentwise as:
\[
\lambda  (\mathbf{A})_{ij} = \lambda a_{ij}
\]
</div>



<div class="example">
Let 
\[
\mathbf{A} =
\begin{pmatrix}
1 & 2 \\
3 & 4
\end{pmatrix}, \quad \text{and let the scalar } k = 3.
\]
Then the scalar multiplication \(k \cdot \mathbf{A}\) is:
\[
3 \cdot 
\begin{pmatrix}
1 & 2 \\
3 & 4
\end{pmatrix} =
\begin{pmatrix}
3 & 6 \\
9 & 12
\end{pmatrix}.
\]
</div>



<div class="lemma">
For \( \mathbf{A}, \mathbf{\mathbf{B}} \in \mathbb{R}^{m \times n} \) and scalars \( \lambda, \psi \in \mathbb{R} \):
\[
(\lambda + \psi)\mathbf{A} = \lambda \mathbf{A} + \psi \mathbf{A}, \quad
\lambda(\mathbf{A} + \mathbf{\mathbf{B}}) = \lambda \mathbf{A} + \lambda \mathbf{\mathbf{B}}.
\]
</div>


---

### Compact Form of Linear Systems


A system of linear equations such as
\[
\begin{aligned}
2x_1 + 3x_2 + 5x_3 &= 1 \\
4x_1 - 2x_2 - 7x_3 &= 8 \\
9x_1 + 5x_2 - 3x_3 &= 2
\end{aligned}
\]
can be written in matrix form as  
\[
\mathbf{A}\mathbf{x} = \mathbf{b},
\quad \text{where} \quad
\mathbf{A} =
\begin{bmatrix}
2 & 3 & 5 \\
4 & -2 & -7 \\
9 & 5 & -3
\end{bmatrix},
\quad
\mathbf{x} =
\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix},
\quad
\mathbf{b} =
\begin{bmatrix}
1 \\ 8 \\ 2
\end{bmatrix}.
\]
This expresses the system as a linear combination of the columns of \( \mathbf{A} \).


<div class="example">
Consider the system of equations:
\[
\begin{cases}
2x + 3y = 5 \\
4x - y = 1
\end{cases}.
\]
We can write this in matrix form as:
\[
\begin{pmatrix}
2 & 3 \\
4 & -1
\end{pmatrix}
\begin{pmatrix}
x \\ y
\end{pmatrix}
=
\begin{pmatrix}
5 \\ 1
\end{pmatrix}.
\]
</div>


<div class="example">
Consider the matrix equation:
\[
\begin{pmatrix}
1 & 2 & -1 \\
3 & 0 & 4
\end{pmatrix}
\begin{pmatrix}
x \\ y \\ z
\end{pmatrix}
=
\begin{pmatrix}
7 \\ 5
\end{pmatrix}.
\]
This corresponds to the system of equations:
\[
\begin{cases}
x + 2y - z = 7 \\
3x + 0 \cdot y + 4z = 5
\end{cases}.
\]
</div>


### Exercises {.unnumbered .unlisted}




<div class="exercise">
Compute $\begin{bmatrix} 2 & -3\\ 1 & 0 \\ -1 & 3\end{bmatrix} + \begin{bmatrix} 9 & -5 \\ 0 & 13 \\ -1 & 3\end{bmatrix}$.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Compute $\begin{bmatrix} 2 & -3\\ 1 & 0 \\ -1 & 3\end{bmatrix} - \begin{bmatrix} 9 & -5 \\ 0 & 13 \\ -1 & 3\end{bmatrix}$.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Compute $\begin{bmatrix} 3 & 6\\2 & 4\end{bmatrix} \begin{bmatrix} 1 & 3\\0 & 2\end{bmatrix}$.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Compute $\begin{bmatrix} 1&2&3\\4&3&2\end{bmatrix} \begin{bmatrix} 2&3\\3&4\\1&2\end{bmatrix}$

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
The textbook talks about $\mathbf{A}\mathbf{B} = \mathbf{C}$, where $c_{ij} = \sum_{l=1}^n a_{il}b_{lj}$.  Use this definition to find $c_{12}$ using this definition if  $\mathbf{A} = \begin{bmatrix} 1&2&3\\4&3&2\end{bmatrix}$ and $\mathbf{B} = \begin{bmatrix} 2&3\\3&4\\1&2\end{bmatrix}$

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Is $\begin{bmatrix} 1&-2\\2&5\end{bmatrix}$ the inverse of $\begin{bmatrix} 5&-2\\2&-1\end{bmatrix}$?
  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
 Prove that the inverse of a matrix $\mathbf{A}$ is unique (if it exists).
 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
 Show that if $\mathbf{A}^{-1}$ exists, then $\det(\mathbf{A}) \neq 0$.
 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Find $\mathbf{A}^{-1}$ and $\mathbf{A}^T$ for $\begin{bmatrix} 2&-1\\-4&3\end{bmatrix}$. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
 Find $\mathbf{A}^{-1}$ and $\mathbf{A}^T$ for $\begin{bmatrix} 3&4/3\\-3&-1\end{bmatrix}$. 
 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Show that, for a 2 x 2 matrix, that $(\mathbf{A}^T)^{-1} = (\mathbf{A}^{-1})^T$. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Show that the sum of symmetric matrices is symmetric. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Find an example where the product of symmetric matrices is not symmetric.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove that if $\mathbf{A}\mathbf{B} = \mathbf{I}$, then $\mathbf{B}\mathbf{A} = \mathbf{I}$.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove Lemma 2.3

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
Prove Lemma 2.4

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove Lemma 2.5

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->




## Solving Systems of Equations



We can represent a system of linear equations as  

\[
\begin{aligned}
a_{11}x_1 + \dots + a_{1n}x_n &= b_1 \\
\vdots \quad\quad \quad\quad \quad\quad \vdots \quad & \quad \vdots \\
a_{m1}x_1 + \dots + a_{mn}x_n &= b_m
\end{aligned}
\]
where \( a_{ij}, b_i \in \mathbb{R} \) are known constants and \( x_j \) are unknowns.  This can be written compactly in matrix form as  
\[
\mathbf{A}\mathbf{x} = \mathbf{b}.
\]

Matrices allow for a compact representation and straightforward manipulation of linear systems.   Next, we focus on finding solutions to these systems and introduce the idea of a **matrix inverse**.

---

### Particular and General Solutions

Consider a system with fewer equations than unknowns. Such systems often have infinitely many solutions.  To solve such a system of equations, follow these steps:


1. Find a particular solution to \( \mathbf{A}\mathbf{x} = \mathbf{b} \).    
2. Find all homogeneous solutions to \( \mathbf{A}\mathbf{x} = \mathbf{0} \).    
3. Combine the results:  
   \[
   \mathbf{x} = \mathbf{x}_p + \mathbf{x}_h.
   \]
Neither the particular solution nor the general solution is unique.


<div class="example">
\[
\begin{bmatrix}
1 & 0 & 8 & -4 \\
0 & 1 & 2 & 12
\end{bmatrix}
\begin{bmatrix}
x_1 \\ x_2 \\ x_3 \\ x_4
\end{bmatrix}
=
\begin{bmatrix}
42 \\ 8
\end{bmatrix}.
\]

From inspection, we can find one **particular solution**:
\[
\mathbf{x}_p =
\begin{bmatrix}
42 \\ 8 \\ 0 \\ 0
\end{bmatrix}.
\]

To find all possible solutions, we add all **non-trivial combinations** that satisfy \( \mathbf{A}\mathbf{x} = \mathbf{0} \).  Let $c_i$ represent column $i$.  Then \[c_3 = 8\begin{bmatrix}1\\0 \end{bmatrix} + 2\begin{bmatrix}0 \\ 1 \end{bmatrix} = 8c_1 + 2 c_2,\] or rather
\[
\begin{aligned}
8c_1 + 2c_2 - c_3 &= 0, \\
-4c_1 + 12c_2 - c_4 &= 0,
\end{aligned}
\]
which yield two **homogeneous solutions**:
\[
\mathbf{x}_1 =
\begin{bmatrix}
8 \\ 2 \\ -1 \\ 0
\end{bmatrix},
\quad
\mathbf{x}_2 =
\begin{bmatrix}
-4 \\ 12 \\ 0 \\ -1
\end{bmatrix}.
\]

Thus, the **general solution** is
\[
\mathbf{x} =
\begin{bmatrix}
42 \\ 8 \\ 0 \\ 0
\end{bmatrix}
+ \lambda_1
\begin{bmatrix}
8 \\ 2 \\ -1 \\ 0
\end{bmatrix}
+ \lambda_2
\begin{bmatrix}
-4 \\ 12 \\ 0 \\ -1
\end{bmatrix},
\quad
\lambda_1, \lambda_2 \in \mathbb{R}.
\]
</div>



In general, systems of equations are not as simple as the example above.  To solve any linear system, we can use **Gaussian elimination**, an algorithmic procedure that:

- Applies **elementary row transformations** to simplify the system, and  
- Transforms it into a form where the three steps above can be applied directly.  

This provides a systematic way to find all solutions to \( \mathbf{A}\mathbf{x} = \mathbf{b} \).



---

### Elementary Transformations



<div class="definition">
**Elementary transformations** are operations that simplify a system of linear equations without changing its solution set.

These are:

1. **Row exchange:** Swap two equations (rows).  
2. **Row scaling:** Multiply an equation (row) by a nonzero scalar \( \lambda \in \mathbb{R} \setminus \{0\} \).  
3. **Row addition:** Add a multiple of one equation (row) to another.
</div>

These operations are key to transforming a system into simpler forms, such as row-echelon form (REF) or reduced row-echelon form (RREF).


<div class="example">
Given a system of equations, 
\begin{align*}
-2x_1 + 4x_2 - 2x_3 - x_4 + 4x_5 &= 3\\
4x_1 - 8x_2 + 3x_3 - 3x_4 + x_5 &= 2\\
x_1 - 2x_2 + x_3 - x_4 + x_5 &= 0\\
x_1 - 2x_2 - 3x_4 + 4x_5 &= a\\
\end{align*}
we write it in **augmented matrix form**:
\[
[\mathbf{A} | \mathbf{b}] =
\begin{bmatrix}
-2 & 4 & -2 & -1 & 4 & -3 \\
4 & -8 & 3 & -3 & 1 & 2 \\
1 & -2 & 1 & -1 & 1 & 0 \\
1 & -2 & 0 & -3 & 4 & a
\end{bmatrix}
\]

By applying elementary row operations, we obtain a simpler (row-echelon) form:
\[
\begin{bmatrix}
1 & -2 & 1 & -1 & 1 & 0 \\
0 & 0 & 1 & -1 & 3 & -2 \\
0 & 0 & 0 & 1 & -2 & 1 \\
0 & 0 & 0 & 0 & 0 & a + 1
\end{bmatrix}
\]

The resulting equations are:
\[
\begin{aligned}
x_1 - 2x_2 + x_3 - x_4 + x_5 &= 0 \\
x_3 - x_4 + 3x_5 &= -2 \\
x_4 - 2x_5 &= 1 \\
0 &= a + 1
\end{aligned}
\]

A **particular solution** exists only when \( a = -1 \):
\[
\mathbf{x}_p =
\begin{bmatrix}
2 \\ 0 \\ -1 \\ 1 \\ 0
\end{bmatrix}
\]

The **general solution** is:
\[
\mathbf{x} =
\begin{bmatrix}
2 \\ 0 \\ -1 \\ 1 \\ 0
\end{bmatrix}
+ \lambda_1
\begin{bmatrix}
2 \\ 1 \\ 0 \\ 0 \\ 0
\end{bmatrix}
+ \lambda_2
\begin{bmatrix}
2 \\ 0 \\ -1 \\ 2 \\ 1
\end{bmatrix}, \quad \lambda_1, \lambda_2 \in \mathbb{R}.
\]
</div>




<div class="definition">
A matrix is in **row-echelon form** if:

1. All-zero rows are at the bottom.  
2. Each pivot (first nonzero entry from the left) is strictly to the right of the pivot in the row above.  
</div>

This produces a *“staircase” structure*.



<div class="definition">
**Basic variables** correspond to pivot columns.  **Free variables** correspond to non-pivot columns.  
</div>

In the previous example: \(x_1, x_3, x_4\) are basic; \(x_2, x_5\) are free.



<div class="definition">
A system is in **reduced row-echelon form** if:

1. It is in REF.  
2. Each pivot is 1.  
3. Each pivot is the only nonzero entry in its column.
</div>

RREF makes it easy to:

- Identify basic and free variables.  
- Read off particular and general solutions.  



<div class="note">
**Gaussian elimination** is a process that systematically applies elementary transformations to bring a matrix to RREF.
</div>

Gaussian elimination allows direct solution of \(\mathbf{A}\mathbf{x} = \mathbf{b}\) or \(\mathbf{A}\mathbf{x} = \mathbf{0}\).



<div class="example">
\[
\mathbf{A} =
\begin{bmatrix}
1 & 3 & 0 & 0 & 3 \\
0 & 0 & 1 & 0 & 9 \\
0 & 0 & 0 & 1 & -4
\end{bmatrix}
\]

Pivot columns: 1, 3, and 4 (basic variables).  
Non-pivot columns:  2 and 5 (free variables).

The general solution to \(\mathbf{A}\mathbf{x} = \mathbf{0}\) is:
\[
\mathbf{x} =
\lambda_1
\begin{bmatrix}
3 \\ -1 \\ 0 \\ 0 \\ 0
\end{bmatrix}
+
\lambda_2
\begin{bmatrix}
3 \\ 0 \\ 9 \\ -4 \\ -1
\end{bmatrix}, \quad
\lambda_1, \lambda_2 \in \mathbb{R}.
\]
</div>

---

### The Minus-1 Trick

For a homogeneous system \(\mathbf{A}\mathbf{x} = \mathbf{0}\) in RREF:

1. Extend \(\mathbf{A}\) to an \(n \times n\) matrix \( \tilde{\mathbf{A}} \) by adding rows with a single **−1** where pivots are missing.  
2. The columns of \( \tilde{\mathbf{A}} \) containing **−1** entries form a basis for the solution space (kernel/null space).  




<div class="example">
For \[
\mathbf{A} =
\begin{bmatrix}
1 & 3 & 0 & 0 & 3 \\
0 & 0 & 1 & 0 & 9 \\
0 & 0 & 0 & 1 & -4
\end{bmatrix},
\]
we use the -1 trick to get
\[
\tilde{\mathbf{A}} =
\begin{bmatrix}
1 & 3 & 0 & 0 & 3 \\
0 & -1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 9 \\
0 & 0 & 0 & 1 & -4 \\
0 & 0 & 0 & 0 & -1
\end{bmatrix}.
\]
Columns 2 and 5 are exactly the solutions for the homogeneous equation.
</div>

The columns with −1 yield the same solution vectors as before.



<div class="example">
Consider the homogeneous system:
\[
\begin{cases}
x + 2y - z = 0 \\
2x + 4y - 2z = 0.
\end{cases}
\]
We can write this as an augmented matrix:
\[
\left[\begin{array}{ccc|c}
1 & 2 & -1 & 0 \\
2 & 4 & -2 & 0
\end{array}\right].
\]
Row reduction gives us
\[
\left[\begin{array}{ccc|c}
1 & 2 & -1 & 0 \\
0 & 0 & 0 & 0
\end{array}\right]
\]
We rewrite with the -1 trick:
\[
\left[\begin{array}{ccc|c}
1 & 2 & -1 & 0 \\
0 & -1 & 0 & 0\\
0 & 0 & -1 & 0\\
\end{array}\right]
\]
Hence the general (homogeneous) solution is:

\[
\mathbf{x} = s\begin{bmatrix} 2 \\ -1 \\ 0 \end{bmatrix} + 
t\begin{bmatrix} -1 \\ 0 \\ -1 \end{bmatrix}, \quad s,t \in \mathbb{R}.
\]
</div>

---

### Calculating an Inverse Matrix via Gaussian Elimination


<div class="lemma">
To find \(\mathbf{A}^{-1}\) for \(\mathbf{A} \in \mathbb{R}^{n \times n}\), use Gaussian elimination to transform:
\[
[\mathbf{A} | \mathbf{I}_n] \;\Rightarrow\; [\mathbf{I}_n | \mathbf{A}^{-1}].
\]
</div>



<div class="example">
\[
\mathbf{A} =
\begin{bmatrix}
1 & 0 & 2 & 0 \\
1 & 1 & 0 & 0 \\
1 & 2 & 0 & 1 \\
1 & 1 & 1 & 1
\end{bmatrix}.
\]
Append the identity matrix.
\[
[\mathbf{A}|\mathbf{I}_4] =
\begin{bmatrix}
1 & 0 & 2 & 0 &1&0&0&0\\
1 & 1 & 0 & 0 &0&1&0&0\\
1 & 2 & 0 & 1 &0&0&1&0\\
1 & 1 & 1 & 1 &0&0&0&1
\end{bmatrix}.
\]

After elimination:
\[
[\mathbf{I}_4|\mathbf{A}^{-1}] =
\begin{bmatrix}
1&0&0&0&  -1 & 2 & -2 & 2 \\
0&1&0&0&  1 & -1 & 2 & -2 \\
0&0&1&0&  1 & -1 & 1 & -1 \\
0&0&0&1&  -1 & 0 & -1 & 2
\end{bmatrix}.
\]
\[
\mathbf{A}^{-1} =
\begin{bmatrix}
-1 & 2 & -2 & 2 \\
1 & -1 & 2 & -2 \\
1 & -1 & 1 & -1 \\
-1 & 0 & -1 & 2
\end{bmatrix}.
\]

Verification:  You should check that
\[
\mathbf{A} \mathbf{A}^{-1} = \mathbf{I}_4.
\]
</div>







### Algorithms for Solving a System of Linear Equations

When solving a system of linear equations \( \mathbf{A}\mathbf{x} = \mathbf{b} \), we typically assume that a solution exists. If not, approximate methods such as **linear regression** (see Chapter 9) are used.

In special cases where \( \mathbf{A} \) is square and invertible, the exact solution can be written as  
\[
\mathbf{x} = \mathbf{A}^{-1}\mathbf{b}.
\]
However, this is often not feasible, so we use the **Moore–Penrose pseudo-inverse**:
\[
\mathbf{x} = (\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{b},
\]
which provides the minimum-norm least-squares solution. Despite being conceptually simple, this approach is computationally expensive and numerically unstable, so it is rarely used in practice.


For large-scale systems, we use **iterative methods**, which gradually improve an approximate solution. These methods include:

- **Stationary iterative methods:** Richardson, Jacobi, Gauss–Seidel, and successive over-relaxation.
- **Krylov subspace methods:** Conjugate gradients, generalized minimal residual (GMRES), and biconjugate gradients.

These methods iteratively update the estimate of the solution according to  
\[
\mathbf{x}^{(k+1)} = \mathbf{C}\mathbf{x}^{(k)} + \mathbf{d},
\]
reducing the residual error \( \|\mathbf{x}^{(k+1)} - \mathbf{x}^*\| \) at each step until convergence to the true solution \( \mathbf{x}^* \).


### Exercises {.unnumbered .unlisted}







<div class="exercise">
Particular/ General solution example.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>



<div class="exercise">
Gaussian Elimination example.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
-1 trick example.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Find the inverse of  $\begin{bmatrix}1&2&3\\2&3&0\\3&0&1\end{bmatrix}$

<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Moore-Penrose pseudo inverse example.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Solve the system of equations \begin{align*}2x + 5y + 2z & =  - 38\\ 3x - 2y + 4z & = 17\\ - 6x + y - 7z & =  - 12\end{align*}

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
Solve the system of equations \begin{align*}2x - 4y + 5z & =  - 33\\ 4x - y & =  - 5\\  - 2x + 2y - 3z & = 19\end{align*}

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
Solve the system of equations \begin{align*}x - y & = 6\\  - 2x + 2y & = 1\end{align*}

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
Solve the system of equations \begin{align*}2x + 5y & =  - 1\\  - 10x - 25y & = 5\end{align*}

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
Find the inverse of  $\begin{bmatrix}1&0&2&3\\2&3&0&1\\0&3&0&1\\1&0&0&1\end{bmatrix}$.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
 Consider a $4 \times 4$ matrix.  How many operations does Gaussian Elimination require?  What about for an $n \times n$ matrix?

<div style="text-align: right;">
[Solution]( )
</div> 
</div>
















<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->







## Vector Spaces

A **vector space** is a structured set of objects called *vectors* that can be added together and scaled by real numbers (scalars), while remaining within the same set. To understand this concept, we first introduce **groups**, which form the foundation for vector space structure.

---

### Groups





<div class="definition">
A **group** is a set \( G \) with an operation \( \otimes \) that satisfies:

1. **Closure:** \( x \otimes y \in G \) for all \( x, y \in G \).  
2. **Associativity:** \( (x \otimes y) \otimes z = x \otimes (y \otimes z) \).  
3. **Neutral element:** There exists \( e \in G \) such that \( x \otimes e = e \otimes x = x \).  
4. **Inverse element:** For every \( x \in G \), there exists \( y \in G \) such that \( x \otimes y = e \).  

If the operation is also **commutative**, the group is called **Abelian**.
</div>

<div class="example">
- \( (\mathbb{Z}, +) \) is an Abelian group.  
- \( (\mathbb{N}_0, +) \) is *not* a group (no inverse elements).  
- \( (\mathbb{R} \setminus \{0\}, \cdot) \) is an Abelian group.  
- \( (\mathbb{R}^{n \times n}, \cdot) \) forms a group only for invertible matrices—called the **general linear group**, denoted \( \mathrm{GL}(n, \mathbb{R}) \). This group is *not* Abelian because matrix multiplication is not commutative.
</div>



<div class="example">
Prove that $\mathrm{GL}(2,\mathbb{R})$ forms a group under multiplication.

To show that $\mathrm{GL}(2,\mathbb{R})$ forms a group, we verify each property:

1. **Closure**  
   Let \(A, B \in \mathrm{GL}(2,\mathbb{R})\). Then both \(A\) and \(B\) are invertible, meaning there exist matrices \(A^{-1}\) and \(B^{-1}\) such that:
\[
AA^{-1} = A^{-1}A = I, \quad BB^{-1} = B^{-1}B = I
\]
We must show that \(AB\) is also invertible.  Consider the product \(B^{-1}A^{-1}\). Then:
\[
(AB)(B^{-1}A^{-1}) = A(BB^{-1})A^{-1} = AIA^{-1} = AA^{-1} = I
\]
and similarly,
\[
(B^{-1}A^{-1})(AB) = B^{-1}(A^{-1}A)B = B^{-1}IB = BB^{-1} = I
\] 
   Hence, \(B^{-1}A^{-1}\) is the inverse of \(AB\).  Therefore, \(AB\) is invertible and \(AB \in \mathrm{GL}(2,\mathbb{R})\).
   *Closure holds.*

2. **Associativity**  
   Matrix multiplication is associative for all \(2 \times 2\) real matrices. Thus, for any \(A, B, C \in \mathrm{GL}(2,\mathbb{R})\):
   \[
   (AB)C = A(BC)
   \]
   *Associativity holds.*


3. **Identity Element**  
   The \(2 \times 2\) identity matrix \(I\) is it's own inverse. It is therefore a  \(2 \times 2\)  invertible matrix and so is in \mathrm{GL}(2,\mathbb{R})\).  
   *Identity exists.*


4. **Inverse Element**  
   By definition, each \(A \in \mathrm{GL}(2,\mathbb{R})\) is invertible. So there exists \(A^{-1}\) such that:
   \[
   AA^{-1} = A^{-1}A = I
   \]
   Moreover, since $A$ is the inverse of $A^{-1}$, we know that $A^{-1}$ is also an invertible  \(2 \times 2\) matrix and so is in \(\mathrm{GL}(2,\mathbb{R})\).
    *Inverses exist.*
    
Therefore, since all conditions are satisfied, $\mathrm{GL}(2,\mathbb{R})$ is a group under matrix multiplication.
</div>






<div class="definition">
A **real-valued vector space** \( V = (V, +, \cdot) \) consists of:

- An **inner operation** (vector addition) \( + : V \times V \to V \).
- An **outer operation** (scalar multiplication) \( \cdot : \mathbb{R} \times V \to V \).

The operations satisfy:

1. \( (V, +) \) is an Abelian group.  
2. **Distributivity:**    
   - \( \lambda \cdot (\mathbf{x} + \mathbf{y}) = \lambda \cdot \mathbf{x} + \lambda \cdot \mathbf{y} \)    
   - \( (\lambda + \psi) \cdot \mathbf{x} = \lambda \cdot \mathbf{x} + \psi \cdot \mathbf{x} \)   
3. **Associativity:** \( \lambda \cdot (\psi \cdot \mathbf{x}) = (\lambda \psi) \cdot \mathbf{x} \)  
4. **Neutral element:** \( \mathbf{1} \cdot \mathbf{x} = \mathbf{x} \)  
</div>


The zero vector \( \mathbf{0} \) acts as the neutral element of addition.



<div class="example">
- \( \mathbb{R}^n \): Vectors added and scaled componentwise.  
- \( \mathbb{R}^{m \times n} \): Matrices added and scaled elementwise.  
- \( \mathbb{C} \): The complex numbers under standard addition and multiplication.
</div>


In notation, vectors are typically written as column vectors  
\[
\mathbf{x} = 
\begin{bmatrix}
x_1 \\ \vdots \\ x_n
\end{bmatrix},
\]
and their transposes \( \mathbf{x}^\top \) are row vectors.

---

### Vector Subspaces



<div class="definition">
A **vector subspace** \( U \subseteq V \) is a subset of a vector space that is itself a vector space under the same operations.  To be a subspace, \( U \) must satisfy:

1. \( \mathbf{0} \in U \)  
2. **Closure under scalar multiplication:** \( \lambda \mathbf{x} \in U \)  
3. **Closure under addition:** \( \mathbf{x} + \mathbf{y} \in U \)  
</div>



<div class="example">
- The set of all solutions to a **homogeneous** system \( \mathbf{A}\mathbf{x} = \mathbf{0} \) is a subspace of \( \mathbb{R}^n \).  
- The set of solutions to an **inhomogeneous** system \( \mathbf{A}\mathbf{x} = \mathbf{b} \) (where \( \mathbf{b} \neq 0 \)) is *not* a subspace.  
- The intersection of any number of subspaces is also a subspace.
</div>


### Exercises {.unnumbered .unlisted}





<div class="exercise">
SHow that each of these is a group:

- \( (\mathbb{Z}, +) \) is an Abelian group.  
- \( (\mathbb{N}_0, +) \) is *not* a group (no inverse elements).  
- \( (\mathbb{R} \setminus \{0\}, \cdot) \) is an Abelian group.  
- \( (\mathbb{R}^{n \times n}, \cdot) \) forms a group only for invertible matrices—called the **general linear group**, denoted \( \mathrm{GL}(n, \mathbb{R}) \). This group is *not* Abelian because matrix multiplication is not commutative.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
SHow that each of these is a vector space:

- \( \mathbb{R}^n \): Vectors added and scaled componentwise.  
- \( \mathbb{R}^{m \times n} \): Matrices added and scaled elementwise.  
- \( \mathbb{C} \): The complex numbers under standard addition and multiplication.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Show each of these is a vector subspace:

- The set of all solutions to a **homogeneous** system \( \mathbf{A}\mathbf{x} = \mathbf{0} \) is a subspace of \( \mathbb{R}^n \).  
- The set of solutions to an **inhomogeneous** system \( \mathbf{A}\mathbf{x} = \mathbf{b} \) (where \( \mathbf{b} \neq 0 \)) is *not* a subspace.  
- The intersection of any number of subspaces is also a subspace.


<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove that $\mathbb{Z}$ is a group under addition. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove that $\mathbb{Z}^+$ is not a group under addition.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove that $\mathbb{R} \setminus \{0\}$ is a group under multiplication.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Let $G$ be the set of matrices of the form $\begin{bmatrix}a&b\\0&c \end{bmatrix}$ where $a,b,c \in \mathbb{R}$ and $ac \not = 0$.  Prove that $G$ forms a subgroup of $G(2\mathbb{R})$. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Let $g$ be an element of a group $G$.  Show that it's inverse is unique.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
 Let $G$ be a group and let $H$ be a non-empty subset of $G$.  Prove the following are equivalent by proving $1 \Rightarrow 3 \Rightarrow 2 \Rightarrow 1$.
        \begin{enumerate}
            \item[1]    $H$ is a subgroup of $G$.
            \item[2]    (a) $x,y \in H \Rightarrow xy \in H \forall x,y$\\
                        (b) $x \in H \Rightarrow x^{-1} \in H$.
            \item[3]    $x,y \in H \Rightarrow xy^{-1} \in H \forall x,y$.
        \end{enumerate} 


<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove that $\mathbb{R}^n$ is a vector space under componentwise addition and scalar multiplication.


<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">
Prove that $\mathbb{R}^{m \times n}$ is a vector space under componentwise addition and scalar multiplication. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>





<div class="exercise">



<div style="text-align: right;">
[Solution]( )
</div> 
</div>


















<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->








## Linear Independence

In a vector space, vectors can be added to each other and scaled by scalars. The *closure property* ensures the result is still within the same vector space.  Some sets of vectors, when added and scaled, can be used to represent every vector in the space (through linear combinations).  These sets form a **basis** for the space. Before defining a basis, we first define linear combinations and linear independence.





<div class="definition">
A vector  
\[
\mathbf{v} = \lambda_1 x_1 + \lambda_2 x_2 + \cdots + \lambda_k x_k
\]  
is called a **linear combination** of the vectors \(x_1, \ldots, x_k\) where \(\lambda_1, \ldots, \lambda_k \in \mathbb{R}\).
</div>

The **zero vector** can always be written as a linear combination by taking all \(\lambda_i = 0\). A **non-trivial linear combination** occurs when at least one \(\lambda_i \ne 0\).





<div class="definition">
A set of vectors \(x_1, \ldots, x_k \in V\) is **linearly dependent** if there exists a non-trivial combination  
  $$
  \lambda_1 x_1 + \cdots + \lambda_k x_k = 0,
  $$
with at least one \(\lambda_i \ne 0\).  A set of vectors is **linearly independent** if the only solution is the trivial one
  $$
  \lambda_1 = \cdots = \lambda_k = 0.
  $$
</div>

Intuitively, linearly independent vectors contain no redundancy.  Removing one changes the span of the set.

<div class="example">
Directions such as *northwest* and *southwest* form linearly independent directions in a 2D plane. A *west* direction can be expressed as a combination of those two, making the set linearly dependent.
</div>


<div class="note">
- Vectors are either linearly dependent or independent — no third case exists.  
- If any vector is zero or if two vectors are identical, the set is dependent.   
- A set is dependent iff one vector is a linear combination of the others.  
- Scalar multiples of a vector cause dependence: if \(x_i = \lambda x_j\), the set is dependent.  
</div>


<div class="example">
Consider the vectors in \(\mathbb{R}^3\):
\[
\mathbf{v}_1 = \begin{pmatrix}1 \\ 2 \\ 3\end{pmatrix}, \quad
\mathbf{v}_2 = \begin{pmatrix}2 \\ 4 \\ 6\end{pmatrix}, \quad
\mathbf{v}_3 = \begin{pmatrix}0 \\ 1 \\ -1\end{pmatrix}
\]

We check if there exist scalars \(c_1, c_2, c_3\), not all zero, such that
\[
c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + c_3 \mathbf{v}_3 = \mathbf{0}.
\]
Notice that \(\mathbf{v}_2 = 2 \mathbf{v}_1\), so we can take
\[
c_1 = 2, \quad c_2 = -1, \quad c_3 = 0
\]

Then:
\[
2\mathbf{v}_1 - 1\mathbf{v}_2 + 0\mathbf{v}_3 = 2\mathbf{v}_1 - 2\mathbf{v}_1 + \mathbf{0} = \mathbf{0}.
\]
Hence, \(\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}\) is **linearly dependent**.
</div>


---

### Gaussian Elimination Method




<div class="theorem">
A set of vectors \( V = \{ \mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_k \} \) in \( \mathbb{R}^n \) is linearly independent if, when the vectors are placed as columns of a matrix \[
\mathbf{A} = [\mathbf{v}_1 \ \mathbf{v}_2 \ \cdots \ \mathbf{v}_k],
\]
and elementary row operations are performed to reduce \( \mathbf{A} \) to row echelon form, each column of \( \mathbf{A} \) contains a pivot (a leading 1 in some row).
</div>

Equivalently, the set \( V \) is linearly independent if and only if the reduced matrix has a pivot in every column.



<div class="note">
To test independence:

1. Write the vectors as columns of a matrix \(\mathbf{A}\).  
2. Perform row reduction to row echelon form.  
   - Pivot columns correspond to independent vectors.  
   - Non-pivot columns can be expressed as combinations of earlier vectors.  
3. The vectors are independent iff every column is a pivot column.
</div>



<div class="example">
Given  
\[
x_1 =
\begin{bmatrix}
1 \\ 2 \\ -3 \\ 4
\end{bmatrix},
\quad
x_2 =
\begin{bmatrix}
1 \\ 1 \\ 0 \\ 2
\end{bmatrix},
\quad
x_3 =
\begin{bmatrix}
-1 \\ -2 \\ 1 \\ 1
\end{bmatrix},
\]
we can put them into a matrix
\[\begin{bmatrix}
1 & 1 & -1\\ 2 & 1 & -2 \\ -3 & 0 & 1 \\ 4 & 2 & 1
\end{bmatrix}.
\]
Row reduction shows all are pivot columns 
\[\begin{bmatrix}
1 & 1 & -1\\ 2 & 1 & -2 \\ -3 & 0 & 1 \\ 4 & 2 & 1
\end{bmatrix} 
\longrightarrow 
\begin{bmatrix}
1 & 1 & -1\\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0
\end{bmatrix}.
\]
Since all the columns are pivot columns, the set is linearly independent.
</div>





<div class="note">
Let \(\mathbf{b}_1, \ldots, \mathbf{b}_k\) be linearly independent, and define \(x_j = \mathbf{B} \mathbf{\lambda}_j\) with \(\mathbf{B} = [\mathbf{b}_1 \cdots \mathbf{b}_k]\).  Then, vectors \(\{\mathbf{x}_1, \ldots, \mathbf{x}_m\}\) are linearly independent iff the coefficient vectors \(\{\mathbf{\lambda}_1, \ldots, \mathbf{\lambda}_m\}\) are linearly independent.
</div>



<div class="example">
Let 
\[
\mathbf{b}_1 = \begin{pmatrix}1 \\ 0 \\ 0\end{pmatrix}, \quad
\mathbf{b}_2 = \begin{pmatrix}0 \\ 1 \\ 0\end{pmatrix}
\] 
be linearly independent vectors in \(\mathbb{R}^3\), and define 
\(\mathbf{B} = [\mathbf{b}_1 \ \mathbf{b}_2]\).

Let the coefficient vectors be:
\[
\mathbf{\lambda}_1 = \begin{pmatrix}1 \\ 2\end{pmatrix}, \quad
\mathbf{\lambda}_2 = \begin{pmatrix}3 \\ 4\end{pmatrix}.
\]

Then define
\[
\mathbf{x}_j = \mathbf{B} \mathbf{\lambda}_j.
\]

Next, we compute \(\mathbf{x}_1\) and \(\mathbf{x}_2\)
\[
\mathbf{x}_1 = \mathbf{B}\mathbf{\lambda}_1 = 
\begin{pmatrix}1 & 0 \\ 0 & 1 \\ 0 & 0\end{pmatrix} 
\begin{pmatrix}1 \\ 2\end{pmatrix} =
\begin{pmatrix}1 \\ 2 \\ 0\end{pmatrix}, 
\quad
\mathbf{x}_2 = \mathbf{B}\mathbf{\lambda}_2 =
\begin{pmatrix}3 \\ 4 \\ 0\end{pmatrix}.
\]

The claim is that \(\mathbf{x}_1\) and \(\mathbf{x}_2\) are linearly independent.  To verify this, check if \(c_1 \mathbf{x}_1 + c_2 \mathbf{x}_2 = \mathbf{0}\) has only the trivial solution.
\[
c_1 \begin{pmatrix}1 \\ 2 \\ 0\end{pmatrix} + 
c_2 \begin{pmatrix}3 \\ 4 \\ 0\end{pmatrix} =
\begin{pmatrix}c_1 + 3c_2 \\ 2c_1 + 4c_2 \\ 0\end{pmatrix} = \mathbf{0}.
\]
Solve:
\[
c_1 + 3c_2 = 0 \quad \text{and} \quad 2c_1 + 4c_2 = 0.
\]
Both equations are consistent and lead to the trivial solution \(c_1 = 0, c_2 = 0\).  Hence, \(\{\mathbf{x}_1, \mathbf{x}_2\}\) is linearly independent.
</div>



<div class="note">
In any vector space \(V\), if there are more vectors than dimensions (\(m > k\)), the vectors are linearly dependent.
</div>





### Exercises {.unnumbered .unlisted}



<div class="exercise">
Determine if the set of vectors is linearly independent:
$$
\begin{aligned}
x_1 &= b_1 - 2b_2 + b_3 - b_4, \\
x_2 &= -4b_1 - 2b_2 + 4b_4, \\
x_3 &= 2b_1 + 3b_2 - b_3 - 3b_4, \\
x_4 &= 17b_1 - 10b_2 + 11b_3 + b_4,
\end{aligned}
$$

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Prove that if any vector is zero or if two vectors are identical, the set is dependent.   

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Prove that a set of vectors is dependent iff one vector is a linear combination of the others.  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Prove that scalar multiples of a vector cause dependence: if \(x_i = \lambda x_j\), the set is dependent. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>



<div class="exercise">
Are the vectors $\left\{\begin{bmatrix}1\\2\\3\end{bmatrix}, \begin{bmatrix}3\\5\\7\end{bmatrix}, \begin{bmatrix}0\\1\\2\end{bmatrix}\right\}$ linearly independent? 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Are the vectors $\left\{\begin{bmatrix}1\\2\\3\end{bmatrix}, \begin{bmatrix}3\\2\\9\end{bmatrix}, \begin{bmatrix}5\\2\\-1\end{bmatrix}\right\}$ linearly independent? 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Are $p(x) = 1 + 3x + 2x^2$, $q(x) = 3 + x + 2x^2$ and $r(x) = 2x + x^2$ linearly independent?

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->




## Basis and Rank

### Generating Set and Basis




<div class="definition">
Let \( V = (V, +, \cdot) \) be a vector space and \( A = \{ \mathbf{x}_1, \ldots, \mathbf{x}_k \} \subseteq V \).  If every vector \( \mathbf{v} \in  \mathbf{V} \) can be written as a linear combination of \( \mathbf{x}_1, \ldots, \mathbf{x}_k \),  then \( A \) is called a **generating set** of \(  \mathbf{V} \).  
</div>

The set of all such linear combinations is called the **span** of \( A \), denoted by:
\[
V = \text{span}[A] = \text{span}[\mathbf{x}_1, \ldots, \mathbf{x}_k].
\]

A generating set spans the entire vector space, meaning every vector can be expressed as a combination of those in the set.




<div class="definition">
A generating set \( A \subseteq  V \) is **minimal** if no smaller subset \( \tilde{A} \subset A \) spans \( V \).  Every **linearly independent generating set** of \( V \) is minimal and is called a **basis** of \( V \).
</div>

Equivalently:

- A basis is a minimal generating set.  
- A basis is also a maximal linearly independent set (adding any new vector makes it dependent).



<div class="theorem">
For a basis \( B = \{\mathbf{b}_1, \ldots, \mathbf{b}_k\} \), every vector \( \mathbf{x} \in V \) can be expressed **uniquely** as:
\[
\mathbf{x} = \sum_{i=1}^{k} \lambda_i \mathbf{b}_i = \sum_{i=1}^{k} \psi_i \mathbf{b}_i,
\]
where \( \lambda_i = \psi_i \) for all \( i = 1, \ldots, k \).
</div>



<div class="example">
The **standard basis** is:
\[
B= \left\{
\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix},
\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix},
\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}
\right\}.
\]

Other valid bases include:
\[
B_1 =
\left\{
\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix},
\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix},
\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}
\right\}, \quad
B_2 =
\left\{
\begin{bmatrix} 0.5 \\ 0.8 \\ 0.4 \end{bmatrix},
\begin{bmatrix} 1.8 \\ 0.3 \\ 0.3 \end{bmatrix},
\begin{bmatrix} -2.2 \\ -1.3 \\ 3.5 \end{bmatrix}
\right\}.
\]
</div>



<div class="note">
Every vector space \( V \) has a basis, but there are usually many possible bases.  All bases of \( V \) contain the same number of vectors, called **basis vectors**.  The **dimension** of \( V \), written \( \text{dim}(V) \), is the number of basis vectors.  If \( U \subseteq V \) is a subspace, then:
\[
\text{dim}(U) \leq \text{dim}(V),
\]
with equality if and only if \( U = V \).
</div>




<div class="note">
To find a basis for a subspace \( U = \text{span}[\mathbf{x}_1, \ldots, \mathbf{x}_m] \subseteq \mathbb{R}^n\):

1. Form a matrix \( \mathbf{A} \) with the spanning vectors as columns.  
2. Compute the row echelon form of \( \mathbf{A} \).  
3. The columns corresponding to pivot positions form a basis for \( U \).
</div>



<div class="example">
Let \( U \subseteq \mathbb{R}^5 \) be spanned by the vectors:
\[
\mathbf{x}_1 =
\begin{bmatrix}
1 \\ 2 \\ -1 \\ -1 \\ -1
\end{bmatrix}, \quad
\mathbf{x}_2 =
\begin{bmatrix}
2 \\ -1 \\ 1 \\ 2 \\ -2
\end{bmatrix}, \quad
\mathbf{x}_3 =
\begin{bmatrix}
3 \\ -4 \\ 3 \\ 5 \\ -3
\end{bmatrix}, \quad
\mathbf{x}_4 =
\begin{bmatrix}
-1 \\ 8 \\ -5 \\ -6 \\ 1
\end{bmatrix}.
\]

After performing Gaussian elimination on the matrix \( [\mathbf{x}_1 \ \mathbf{x}_2 \ \mathbf{x}_3 \ \mathbf{x}_4] \), the pivot columns correspond to \( \mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_4 \), which are linearly independent. 
Hence, \( \{\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_4\} \) is a basis of \( U \) and $\text{dim}(U) = 3$.  Note also that the standard basis for $\mathbb{R}^5$ has 5 vectors (each one with 0's in each entry other than the $n^{th}$ spot).  So, $\text{dim}(U) < \text{dim}(V)$.
</div>

---

### Rank




<div class="definition">
The **rank** of a matrix \( \mathbf{A} \in \mathbb{R}^{m \times n} \), denoted \( \text{rk}(\mathbf{A}) \), is the number of linearly independent columns (or rows) of \( \mathbf{A} \).  
</div>


<div class="definition">
Let \( \mathbf{A} \in \mathbb{R}^{m \times n} \) be a matrix. The **column space** of \( \mathbf{A} \), denoted as \( \text{Col}(\mathbf{A}) \), is the subspace of \( \mathbb{R}^m \) spanned by the columns of \( \mathbf{A} \).  
</div>

In other words,
  \[
  \text{Col}(\mathbf{A}) = \text{span} \{ \mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_n \},
  \]
where \( \mathbf{a}_i \) is the \( i^\text{th} \) column of \( \mathbf{A} \).  The column space represents all possible vectors \( \mathbf{b} \) for which the linear system \( \mathbf{A}\mathbf{x} = \mathbf{b} \) has a solution.




<div class="definition">
The **row space** of \( \mathbf{A} \), denoted as \( \text{Row}(\mathbf{A}) \), is the subspace of \( \mathbb{R}^n \) spanned by the rows of \( \mathbf{A} \).  
</div>

Equivalently,
  \[
  \text{Row}(\mathbf{A}) = \text{span} \{ \mathbf{r}_1, \mathbf{r}_2, \ldots, \mathbf{r}_m \},
  \]
where \( \mathbf{r}_i \) is the \( i^\text{th} \) row of \( \mathbf{A} \).  The row space contains all possible linear combinations of the rows of \( \mathbf{A} \).

<div class="lemma">
**Properties of the rank:**
Let $\mathbf{A} \in \mathbb{R}^{m \times n}$ matrix and $\mathbf{x}$ and $\mathbf{b}$ vectors of length $n$.

- \( \text{rk}(\mathbf{A}) = \text{rk}(\mathbf{A}^\top) \): column rank equals row rank.  
- The **column space** of \( \mathbf{A} \) has dimension \( \text{rk}(\mathbf{A}) \).  
- The **row space** of \( \mathbf{A} \) also has dimension \( \text{rk}(\mathbf{A}) \).  
- \( \mathbf{A} \) is **invertible** iff \( \text{rk}(\mathbf{A}) = n \) (for \( \mathbf{A} \in \mathbb{R}^{n \times n} \)).  
- The system \( \mathbf{A}\mathbf{x} = \mathbf{b} \) has a solution iff \( \text{rk}(\mathbf{A}) = \text{rk}([\mathbf{A}|\mathbf{b}]) \).  
- The **null space** of \( \mathbf{A} \) (solutions of \( \mathbf{A}\mathbf{x} = 0 \)) has dimension \( n - \text{rk}(\mathbf{A}) \).  
- A matrix has **full rank** if \( \text{rk}(\mathbf{A}) = \min(m, n) \).  
- If \( \text{rk}(\mathbf{A}) < \min(m, n) \), it is **rank deficient**.
</div>



<div class="example"> 

\[
\mathbf{A} =
\begin{bmatrix}
1 & 0 & 1 \\
0 & 1 & 1 \\
0 & 0 & 0
\end{bmatrix}
\Rightarrow \text{rk}(\mathbf{A}) = 2.
\]

\[
\mathbf{A} =
\begin{bmatrix}
1 & 2 & 1 \\
-2 & -3 & 1 \\
3 & 5 & 0
\end{bmatrix}
\Rightarrow
\begin{bmatrix}
1 & 2 & 1 \\
0 & 1 & 3 \\
0 & 0 & 0
\end{bmatrix}
\Rightarrow \text{rk}(\mathbf{A}) = 2.
\]
</div>


### Exercises {.unnumbered .unlisted}




<div class="exercise">
Prove that \( \text{rk}(\mathbf{A}) = \text{rk}(\mathbf{A}^\top) \): column rank equals row rank.  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
Prove that the **column space** of \( \mathbf{A} \) has dimension \( \text{rk}(\mathbf{A}) \).  Prove that the **row space** of \( \mathbf{A} \) also has dimension \( \text{rk}(\mathbf{A}) \).  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
Prove that \( \mathbf{A} \) is **invertible** iff \( \text{rk}(\mathbf{A}) = n \) (for \( \mathbf{A} \in \mathbb{R}^{n \times n} \)).  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
Prove that the system \( \mathbf{A}\mathbf{x} = \mathbf{b} \) has a solution iff \( \text{rk}(\mathbf{A}) = \text{rk}([\mathbf{A}|\mathbf{b}]) \).  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
Prove that the null space of \( \mathbf{A} \) (solutions of \( \mathbf{A}\mathbf{x} = 0 \)) has dimension \( n - \text{rk}(\mathbf{A}) \).  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Let $\mathbf{A}$ be a regular $2 \times 2$ (invertible) matrix.  Show that $\mathbf{A}$ has rank 2.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
 In $\mathbb{R}^{2 \times 2}$, show that $rk(\mathbf{A}) = 1$ if $\det(\mathbf{A}) = 0$, but $\mathbf{A}$ is not the zero matrix.
 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
Find the rank of $\mathbf{A} = \begin{bmatrix}1&0&2&1\\ 0&2&4&2\\0&2&2&1 \end{bmatrix}$. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
Find the rank of $\mathbf{A} = \begin{bmatrix}1&2&1&-1\\ 9&5&2&2\\ 7&1&0&4 \end{bmatrix}$.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">
Let $\mathbf{x}$ be a unit vector in $\mathbb{R}^n$.  Partition $\mathbf{x}$ as \[\mathbf{x} = \begin{bmatrix}x_1 \\ x_2 \\ \vdots \\x_n \end{bmatrix} = \begin{bmatrix} x_1 \\ \mathbf{y} \end{bmatrix}.\]  Let \[\mathbf{Q} = \begin{bmatrix}x_1 & \mathbf{y}^T\\ \mathbf{y} & \mathbf{I} - \left(\dfrac{1}{1 - x_1} \right) \mathbf{y} \mathbf{y}^T \end{bmatrix}.\]  $Q$ is orthogonal.  (This procedure gives a quick method for finding an orthonormal basis for $\mathbb{R}^n$ with a prescribed first vector $\mathbf{x}$, a construction that is frequently useful in applications).  Select any non-trivial vector in $\mathbb{R}^3$ and verify that $\mathbf{Q}$ is orthogonal. 

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->




## Linear Mappings



Linear mappings are functions between vector spaces that preserve vector addition and scalar multiplication. That is, for vector spaces \( V \) and \( W \), a mapping \( \Phi : V \to W \) is *linear* if:
\[
\Phi(x + y) = \Phi(x) + \Phi(y), \quad \Phi(\lambda x) = \lambda \Phi(x)
\]
for all \( x, y \in V \) and scalars \( \lambda \in \mathbb{R} \).




<div class="definition">
A **linear mapping** (or **linear transformation**) is a function \( \Phi : V \to W \) satisfying:
\[
\Phi(\lambda x + \psi y) = \lambda \Phi(x) + \psi \Phi(y)
\]
for all \( x, y \in V \) and scalars \( \lambda, \psi \in \mathbb{R} \).
</div>



<div class="example">
A linear mapping \( T: \mathbb{R}^2 \to \mathbb{R}^2 \) can be represented by a matrix.  Consider the matrix
\[
\mathbf{A} = \begin{pmatrix}
2 & -1 \\
3 & 4
\end{pmatrix}.
\]
Define the linear map \( \Phi(\mathbf{x}) = \mathbf{A}\mathbf{x} \).

Let  
\[
\mathbf{x} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}.
\]
Then
\[
\Phi(\mathbf{x})
= A\mathbf{x}
= \begin{pmatrix}
2 & -1 \\
3 & 4
\end{pmatrix}
\begin{pmatrix}
1 \\ 2
\end{pmatrix}
=
\begin{pmatrix}
2(1) - 1(2) \\
3(1) + 4(2)
\end{pmatrix}
=
\begin{pmatrix}
0 \\ 11
\end{pmatrix}.
\]

The map is linear because \[\Phi(\mathbf{v} + \mathbf{w}) = \mathbf{A}(\mathbf{v} + \mathbf{w}) = \mathbf{A}\mathbf{v} + \mathbf{A}\mathbf{w} = \Phi(\mathbf{v}) + \Phi(\mathbf{w}) \quad \text{and} \quad \phi(\lambda \mathbf{v}) = \mathbf{A}(\lambda \mathbf{v}) = \lambda \mathbf{A}\mathbf{v} = \lambda \Phi(\mathbf{v}).\]
</div>


Injective, surjective and bijective mappings are important in machine learning.  Bijective maps are particularly useful because they are invertible.


<div class="definition">
Let \( \Phi : V \to W \) be a mapping.  Then

-  $\Phi$ is **injective** if \( \Phi(x) = \Phi(y) \Rightarrow x = y \).  
-  $\Phi$ is **surjective** if \( \Phi(V) = W \).  
-  $\Phi$ is **bijective** if it is both injective and surjective.  
</div>




<div class="example">
The linear map \( \Phi(\mathbf{x}) = \mathbf{A}\mathbf{x} \), with
\[
\mathbf{A} = \begin{pmatrix}
2 & -1 \\
3 & 4
\end{pmatrix}
\]
is bijective.  One way you can determine if a function is invertible is to determine if it can be inverted.  In this case, we can find the inverse function by multiplying by the inverse of $\mathbf{A}$ (if it exists).  Since the determinant of $\mathbf{A} \not = 0$, the matrix (and hence the function) is invertible.  Therefore, the function is bijective.
</div>



<div class="note">
Some other special cases of linear mappings include:

- **Isomorphism:** Linear and bijective (\( \Phi : V \to W \))  
- **Endomorphism:** Linear map from \( V \) to itself (\( \Phi : V \to V \))  
- **Automorphism:** Linear and bijective endomorphism  
</div>






<div class="example">
The function
\[
f: \mathbb{R} \to \mathbb{R}, \qquad f(x) = 3x - 2,
\]
is a bijection from $\mathbb{R}$ to $\mathbb{R}$.  Since it is also linear, it is an automorphism on $\mathbb{R}$.  

The function
\[
g: \mathbb{R} \to \mathbb{R}, \qquad g(x) = x^2,
\]
is not a bijection.  In fact, it is neither injective or surjective.  It's not injective since
\[
g(2) = 4, \qquad g(-2) = 4.
\]
Since  
\[
2 \neq -2 \quad \text{but} \quad g(2) = g(-2),
\]
the function is not injective.  Also, since the codomain is \(\mathbb{R}\), but
\(g(x) = x^2\) only produces nonnegative outputs, negative numbers (e.g. \(-5\)) cannot be written as \(x^2\) for any real \(x\). Thus it does not hit every value in the codomain.  Therefore, it is not surjective.
</div>

The **identity mapping** is denoted \( \text{id}_V(x) = x \).




<div class="theorem">
Finite-dimensional vector spaces \( V \) and \( W \) are isomorphic if and only if:
\[
\dim(V) = \dim(W)
\]
</div>

This means that spaces of equal dimension are structurally the same, as they can be related through a linear bijective map.



### Matrix Representation of Linear Mappings

Every linear mapping between finite-dimensional vector spaces can be represented by a matrix.  


<div class="lemma">
Given bases \( \mathbf{B} = (\mathbf{b}_1, \dots, \mathbf{b}_n) \) for \( V \) and \( \mathbf{C} = (\mathbf{c}_1, \dots, \mathbf{c}_m) \) for \( W \), the **transformation matrix** \( \mathbf{A}_\Phi \) of \( \Phi : V \to W \) is defined by:
\[
\Phi(\mathbf{b}_j) = \sum_{i=1}^m \alpha_{ij} \mathbf{c}_i
\]
where \( \mathbf{A}_\Phi(i, j) = \alpha_{ij} \).
</div>

For coordinate vectors \( \hat{\mathbf{x}}  \in V\) and \( \hat{\mathbf{y}} \in W\),
\[
\hat{\mathbf{y}} = \mathbf{A}_\Phi \hat{\mathbf{x}}
\]




<div class="example">
Let

- \(V = \mathbb{R}^2\) with **standard basis**  
  \[
  \mathbf{B} = (\mathbf{b}_1, \mathbf{b}_2)
  = 
  \left(
  \begin{bmatrix}1 \\ 0\end{bmatrix},
  \begin{bmatrix}0 \\ 1\end{bmatrix}
  \right)
  \]

- \(W = \mathbb{R}^2\) with a **nonstandard basis**  
  \[
  \mathbf{C} = (\mathbf{c}_1, \mathbf{c}_2)
  =
  \left(
  \begin{bmatrix}2 \\ 1\end{bmatrix},
  \begin{bmatrix}1 \\ 3\end{bmatrix}
  \right).
  \]

Suppose the linear transformation \( \Phi : \mathbb{R}^2 \to \mathbb{R}^2 \) is
\[
\Phi(x,y) = (3x + y, \; x - 2y).
\]

Since \(\mathbf{B}\) is the standard basis, the inputs are easy:
\begin{align*} 
\mathbf{b}_1 = (1,0) \qquad & \Longrightarrow \qquad \Phi(\mathbf{b}_1)
  = \Phi(1,0)
  = (3, \; 1)\\
\mathbf{b}_2 = (0,1) \qquad & \Longrightarrow \qquad \Phi(\mathbf{b}_2)
  = \Phi(0,1)
  = (1, \; -2)
\end{align*}

These must now be written in basis **C**.

We want to find scalars \( \alpha_{ij} \) such that
\[
\Phi(\mathbf{b}_j)
= \alpha_{1j}\mathbf{c}_1 + \alpha_{2j}\mathbf{c}_2.
\]

For \( \Phi(\mathbf{b}_1) = (3,1) \), solve:
\[
\alpha_{11}\begin{bmatrix}2\\1\end{bmatrix}
+
\alpha_{21}\begin{bmatrix}1\\3\end{bmatrix}
=
\begin{bmatrix}3\\1\end{bmatrix} \qquad \Longrightarrow \qquad
\alpha_{11} = \frac{8}{5}, \qquad \alpha_{21} = -\frac{1}{5}.
\]

For \( \Phi(\mathbf{b}_2) = (1,-2) \), we solve:
\[
\alpha_{12}\begin{bmatrix}2\\1\end{bmatrix}
+
\alpha_{22}\begin{bmatrix}1\\3\end{bmatrix}
=
\begin{bmatrix}1\\ -2\end{bmatrix} \qquad \Longrightarrow \qquad
\alpha_{12} = \frac{5}{7}, \qquad \alpha_{22} = -\frac{3}{7}.
\]

Therefore, the transformation matrix is (by the lemma):
\[
A_\Phi(i,j) = \alpha_{ij} \qquad \Longrightarrow \qquad
A_\Phi^{\mathbf{C}\leftarrow\mathbf{B}}
=
\begin{bmatrix}
\frac{8}{5} & \frac{5}{7} \\
-\frac{1}{5} & -\frac{3}{7}
\end{bmatrix}.
\]

This matrix converts **B-coordinates of \(x\)** into **C-coordinates of \( \Phi(x) \)**:

\[
[\Phi(x)]_{\mathbf{C}}
=
A_\Phi^{\mathbf{C}\leftarrow\mathbf{B}} \,[x]_{\mathbf{B}}.
\]

For an example with a specific vector, let
\[
[x]_{\mathbf{B}} = \begin{bmatrix}2 \\ -1\end{bmatrix}.
\]

Compute:
\[
[\Phi(x)]_{\mathbf{C}}
=
\begin{bmatrix}
\frac{8}{5} & \frac{5}{7} \\
-\frac{1}{5} & -\frac{3}{7}
\end{bmatrix}
\begin{bmatrix}2\\ -1\end{bmatrix}
=
\begin{bmatrix}
\frac{16}{5} - \frac{5}{7} \\[4pt]
-\frac{2}{5} + \frac{3}{7}
\end{bmatrix}
=
\begin{bmatrix}
\frac{87}{35} \\[4pt]
\frac{1}{35}
\end{bmatrix}.
\]

So the coordinate vector of \( \Phi(x) \) **in basis C** is:
\[
[\Phi(x)]_{\mathbf{C}} = \begin{bmatrix}87/35 \\ 1/35\end{bmatrix}.
\]
</div>

If we simply want to convert a vector from $\mathbf{B}$ coordinates to $\mathbf{C}$ coordinates, use the same process with $\Phi(\mathbf{x}) = \mathbf{x}$ (that is, omit the beginning portion of the calculation).



### Coordinate Systems and Bases

A basis defines a coordinate system for a vector space.  



<div class="definition">
Given an ordered basis \( \mathbf{B} = ( \mathbf{b}_1, \dots,  \mathbf{b}_n) \), any vector \( x \in V \) can be uniquely represented as:
\[
 \mathbf{x} = \alpha_1  \mathbf{b}_1 + \cdots + \alpha_n  \mathbf{b}_n
\]
The vector \( \alpha = [\alpha_1, \dots, \alpha_n]^T \) is the **coordinate vector** of \( x \) with respect to \( \mathbf{B} \).
</div>

<div class="example">
Let  
\[
\mathbf{B} = 
(\mathbf{b}_1, \mathbf{b}_2)
=
\left(
\begin{bmatrix}1 \\ 2\end{bmatrix},
\begin{bmatrix}3 \\ 1\end{bmatrix}
\right).
\]

Let
\[
\mathbf{x} = \begin{bmatrix}7 \\ 5\end{bmatrix}.
\]
We want to express \( \mathbf{x} \) as a unique linear combination of the basis vectors:
\begin{align*}
\mathbf{x} &= \alpha_1 \mathbf{b}_1 + \alpha_2 \mathbf{b}_2\\
\begin{bmatrix}7 \\ 5\end{bmatrix} &= \alpha_1 \begin{bmatrix}1 \\ 2\end{bmatrix} + \alpha_2 \begin{bmatrix}3 \\ 1\end{bmatrix}.
\end{align*}


This gives the system:

\[
\begin{cases}
\alpha_1 + 3\alpha_2 = 7 \\
2\alpha_1 + \alpha_2 = 5
\end{cases}
\]

Solving:
\[
\alpha_1 = \frac{8}{5}, \qquad
\alpha_2 = \frac{9}{5} \qquad \Longrightarrow \qquad
[\mathbf{x}]_{\mathbf{B}}
=
\begin{bmatrix}
8/5 \\[4pt]
9/5
\end{bmatrix}.
\]

So, the coordinate vector of $\mathbf{x} = \begin{bmatrix}7 \\ 5\end{bmatrix}$ with respect to the basis $\mathbf{B} = \left(\begin{bmatrix}1 \\ 2\end{bmatrix}, \begin{bmatrix}3 \\ 1\end{bmatrix}\right)$ is \[[\mathbf{x}]_{\mathbf{B}} = \begin{bmatrix} 8/5 \\ 9/5 \end{bmatrix}.\]
</div>


---

### Basis Change and Equivalence

When the bases of \( V \) and \( W \) are changed, the transformation matrix changes accordingly.  




<div class="note">
If \( S \) and \( T \) are the change-of-basis matrices for \( V \) and \( W \), then:
\[
\tilde{\mathbf{A}}_\Phi = T^{-1} \mathbf{A}_\Phi S
\]
</div>


<div class="example">
Let \(V=W=\mathbb{R}^2\) and let \(\Phi:V\to W\) be the linear map whose matrix in the standard basis is
\[
\mathbf{A} =
\begin{bmatrix}
2 & 1\\[4pt]
0 & 3
\end{bmatrix},
\qquad
\text{i.e. }\; \Phi(x)=\mathbf{A}x.
\]

Now pick new ordered bases for \(V\) and \(W\):

- New basis of \(V\) (columns of \(S\)):
  \[
  \mathbf{B} = (b_1,b_2),\qquad
  S = [\,b_1\; b_2\,] =
  \begin{bmatrix}
  1 & 1\\
  1 & -1
  \end{bmatrix}.
  \]
  (So \(b_1=(1,1)^\top,\; b_2=(1,-1)^\top\).)

- New basis of \(W\) (columns of \(T\)):
  \[
  \mathbf{C} = (c_1,c_2),\qquad
  T = [\,c_1\; c_2\,] =
  \begin{bmatrix}
  2 & 0\\
  0 & 3
  \end{bmatrix}.
  \]
  (So \(c_1=(2,0)^\top,\; c_2=(0,3)^\top\).)

Recall the change-of-basis formula:
\[
\widetilde{\mathbf{A}}_\Phi \;=\; T^{-1}\,\mathbf{A}_\Phi\,S,
\]
where \(\widetilde{\mathbf{A}}_\Phi\) is the matrix of \(\Phi\) in the new bases \(\mathbf{B},\mathbf{C}\).

Compute \(\mathbf{A}S\):
\[
\mathbf{A}S =
\begin{bmatrix}2&1\\[4pt]0&3\end{bmatrix}
\begin{bmatrix}1&1\\[4pt]1&-1\end{bmatrix}
=
\begin{bmatrix}3 & 1\\[4pt]3 & -3\end{bmatrix}.
\]

Compute \(T^{-1}\) and then \(\widetilde{\mathbf{A}}_\Phi\):
\[
T^{-1} = \begin{bmatrix}1/2 & 0\\[4pt]0 & 1/3\end{bmatrix},\qquad
\widetilde{\mathbf{A}}_\Phi
= T^{-1}(\mathbf{A}S)
=
\begin{bmatrix}1/2 & 0\\[4pt]0 & 1/3\end{bmatrix}
\begin{bmatrix}3 & 1\\[4pt]3 & -3\end{bmatrix}
=
\begin{bmatrix}3/2 & 1/2\\[4pt]1 & -1\end{bmatrix}.
\]

So in the new bases \(\mathbf{B},\mathbf{C}\) the transformation matrix is
\[

\widetilde{\mathbf{A}}_\Phi =
\begin{bmatrix}3/2 & 1/2\\[4pt]1 & -1\end{bmatrix}.
\]


We can check the same result by mapping the new domain basis vectors and expressing the results in the new codomain basis:

- \( \Phi(b_1) = \mathbf{A} b_1 = \mathbf{A}\begin{bmatrix}1\\[2pt]1\end{bmatrix} = \begin{bmatrix}3\\[2pt]3\end{bmatrix}.\)  
  Solve \(\begin{bmatrix}3\\3\end{bmatrix} = \alpha_1 c_1 + \alpha_2 c_2 = \alpha_1\begin{bmatrix}2\\0\end{bmatrix}+\alpha_2\begin{bmatrix}0\\3\end{bmatrix}\).  
  This gives \(\alpha_1=3/2,\; \alpha_2=1\). So the first column of \(\widetilde{\mathbf{A}}_\Phi\) is \(\begin{bmatrix}3/2\\[2pt]1\end{bmatrix}\).

- \( \Phi(b_2) = \mathbf{A} b_2 = \mathbf{A}\begin{bmatrix}1\\[2pt]-1\end{bmatrix} = \begin{bmatrix}1\\[2pt]-3\end{bmatrix}.\)  
  Solve \(\begin{bmatrix}1\\-3\end{bmatrix} = \beta_1 c_1 + \beta_2 c_2\).  
  This gives \(\beta_1=1/2,\; \beta_2=-1\). So the second column is \(\begin{bmatrix}1/2\\[2pt]-1\end{bmatrix}\).

These columns match \(\widetilde{\mathbf{A}}_\Phi\) above, confirming
\[
\widetilde{\mathbf{A}}_\Phi = T^{-1}\mathbf{A}S.
\]

</div>



### Image and Kernel of a Linear Mapping

The **image** and **kernel** are important subspaces associated with a linear mapping.


<div class="definition">
For a linear mapping \( \Phi : V \to W \), the **kernel / null space** of a mapping is the set of values that map to $0_W \in W$.
\[
\ker(\Phi) := \{ \mathbf{v} \in V : \Phi(\mathbf{v}) = \mathbf{0}_W \}.
\]
The **image / range** is the set of value which get mapped to.  
\[
\mathrm{Im}(\Phi) := \{ \mathbf{w} \in W : \exists \mathbf{v} \in \mathbf{V}, \Phi(\mathbf{v}) = \mathbf{w} \}
\]
</div>



<div class="note">
- \(\mathbf{0}_V \in \ker(\Phi)\), so the null space is never empty.  
- \(\ker(\Phi) \subseteq V\) and \(\mathrm{Im}(\Phi) \subseteq W\) are subspaces.  
- \(\Phi\) is **injective** if and only if \(\ker(\Phi) = \{\mathbf{0}\}\).  
</div>






<div class="definition">
For a matrix \(\mathbf{A} \in \mathbb{R}^{m \times n}\) representing \(\Phi : \mathbb{R}^n \to \mathbb{R}^m\), \(x \mapsto \mathbf{A}x\):

- **Image / Column space**
\[
\mathrm{Im}(\Phi) = \mathrm{span}\{\mathbf{a}_1, \dots, \mathbf{a}_n\} \subseteq \mathbb{R}^m
\]
where \(\mathbf{a}_i\) are the columns of \(\mathbf{A}\).

- **Kernel / Null space**
\[
\ker(\Phi) = \{ \mathbf{x} \in \mathbb{R}^n : \mathbf{A}\mathbf{x} = \mathbf{0} \} \subseteq \mathbb{R}^n
\]
represents all linear combinations of columns that yield zero.
</div>





<div class="example">
Consider \(\Phi : \mathbb{R}^4 \to \mathbb{R}^2\) with

\[
\mathbf{A} = 
\begin{bmatrix}
1 & 2 & -1 & 0 \\
1 & 0 & 0 & 1
\end{bmatrix}, 
\quad
\Phi(\mathbf{x}) = \mathbf{A}\mathbf{x}
\]

- **Image:**
\[
\mathrm{Im}(\Phi) = \mathrm{span}\left\{
\begin{bmatrix}1\\1\end{bmatrix}, 
\begin{bmatrix}2\\0\end{bmatrix}, 
\begin{bmatrix}-1\\0\end{bmatrix}, 
\begin{bmatrix}0\\1\end{bmatrix} \right\}
\]

- **Kernel:**
\[
\ker(\Phi) = \mathrm{span}\left\{
\begin{bmatrix}0\\1/2\\1\\0\end{bmatrix}, 
\begin{bmatrix}-1\\1/2\\0\\1\end{bmatrix} \right\}
\]
</div>


The theorem below is known as the **Rank-Nullity Theorem**.

<div class="theorem">
For \(\Phi : V \to W\):
\[
\dim(\ker(\Phi)) + \dim(\mathrm{Im}(\Phi)) = \dim(V)
\]
</div>





<div class="corollary">
For \(\Phi : V \to W\), the following two facts are true:

1.  If \(\dim(\mathrm{Im}(\Phi)) < \dim(V)\), then \(\ker(\Phi)\) is non-trivial (\(\dim(\ker(\Phi)) \ge 1\)).  

2.  If \(\dim(V) = \dim(W)\), then
\[
\Phi \text{ injective } \iff \Phi \text{ surjective } \iff \Phi \text{ bijective.}
\]
</div>







### Exercises {.unnumbered .unlisted}


<div class="exercise">
Consider \(\Phi : \mathbb{R}^4 \to \mathbb{R}^2\) with

\[
\mathbf{A} = 
\begin{bmatrix}
1 & 2 & -1 & 0 \\
1 & 0 & 0 & 1
\end{bmatrix}, 
\quad
\Phi(\mathbf{x}) = \mathbf{A}\mathbf{x}
\]
Verify that

- **Image:**
\[
\mathrm{Im}(\Phi) = \mathrm{span}\left\{
\begin{bmatrix}1\\1\end{bmatrix}, 
\begin{bmatrix}2\\0\end{bmatrix}, 
\begin{bmatrix}-1\\0\end{bmatrix}, 
\begin{bmatrix}0\\1\end{bmatrix} \right\}
\]

- **Kernel:**
\[
\ker(\Phi) = \mathrm{span}\left\{
\begin{bmatrix}0\\1/2\\1\\0\end{bmatrix}, 
\begin{bmatrix}-1\\1/2\\0\\1\end{bmatrix} \right\}
\]


<div style="text-align: right;">
[Solution]( )
</div> 
</div>



<div class="exercise">
Prove that for \(\Phi : V \to W\), the following two facts are true:

1.  If \(\dim(\mathrm{Im}(\Phi)) < \dim(V)\), then \(\ker(\Phi)\) is non-trivial (\(\dim(\ker(\Phi)) \ge 1\)).  

2.  If \(\dim(V) = \dim(W)\), then
\[
\Phi \text{ injective } \iff \Phi \text{ surjective } \iff \Phi \text{ bijective.}
\]

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Prove the Rank-Nullity Theorem.

<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Let $f: \bbR \rightarrow \bbR$ be defined as $f(x) = x^3$.  Show that $f$ is bijective.  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Let $f: \bbR \rightarrow \bbR$ be defined as $f(x) = x^2$.  Show that $f$ is not bijective. 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Let $f: \bbR^+ \rightarrow \bbR^+$ be defined as $f(x) = \sqrt{x}$.  Show that $f$ is bijective.  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Show that there is an isomorphism between $S = \set{1,2,3,4,5}$ and $\bbZ_5 = \set{0,1,2,3,4}$.  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Write the coordinate $\begin{bmatrix}2\\3\end{bmatrix}$ in terms of the standard basis vectors in $\bbR^2$.  Then write it in terms of the basis $\bb_1=\begin{bmatrix}1\\-1\end{bmatrix}$ and $\bb_2=\begin{bmatrix}1\\1\end{bmatrix}$.  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Write the coordinate $\begin{bmatrix}4\\-1\end{bmatrix}$ in terms of the standard basis vectors in $\bbR^2$.  Then write it in terms of the basis $\bb_1=\begin{bmatrix}0\\-1\end{bmatrix}$ and $\bb_2=\begin{bmatrix}-1\\0\end{bmatrix}$.  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
 On graph paper, create a grid with a standard basis.  Then, in a different colour, create a grid using the basis $\bb_1 = \begin{bmatrix}2\\1\end{bmatrix}$ and $\bb_2 = \begin{bmatrix}-1\\1\end{bmatrix}$.  If $\ba = \begin{bmatrix}1\\3\end{bmatrix}$ is a vector in the second basis, what are the coordinates of that vector in the standard basis.  Draw the vector on the grid.  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Prove that if $V$ is a vector space with basis $\{\bb_1, \ldots, \bb_n\}$, then every vector $\mathbf{v} \in V$ can be written uniquely as a linear combination of $\bb_1, \ldots , \bb_n$.  
<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
 Find the change of basis matrix from $B$ to $C$ and from $C$ to $B$.  Show that they are inverses of each other. \\
       Here, $V = \bbR^2$; $B = \set{\begin{bmatrix}9\\2\end{bmatrix}, \begin{bmatrix}4\\−3\end{bmatrix}}$; $C = \set{\begin{bmatrix}2\\1\end{bmatrix}, \begin{bmatrix}−3\\1\end{bmatrix}}$. 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Find the change of basis matrix from $B$ to $C$ and from $C$ to $B$.  Show that they are inverses of each other.  \\
       Here, $V = \bbR^3$; $B=\set{\begin{bmatrix}2\\−5\\0\end{bmatrix},\begin{bmatrix}3\\0\\5\end{bmatrix},
       \begin{bmatrix}8\\−2\\−9\end{bmatrix}}; C=\set{\begin{bmatrix}1\\−1\\1\end{bmatrix}, \begin{bmatrix}2\\0\\1\end{bmatrix},\begin{bmatrix}0\\1\\3\end{bmatrix}}$. 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
Define $\Phi(\bA) = \bA + \bA^T$.  What is the $\ker(\Phi)$? 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Define $\Phi(\bA) = \begin{bmatrix}0&1\\0&0 \end{bmatrix}\bA$.  What is the $\ker(\Phi)$? 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Define $\Phi: \bbR^3 \rightarrow \bbR^4$ given by $\Phi\left(\begin{bmatrix}x\\y\\z\end{bmatrix} \right) = \begin{bmatrix}x\\x\\y\\y\end{bmatrix}$.  What is the $\ker(\Phi)$? 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<div class="exercise">
 Define $\Phi: \bbR^{2 \times 2} \rightarrow \bbR^{2 \times 2}$ given by $\Phi\left(\begin{bmatrix}a & b \\ c & d\end{bmatrix}\right) = \begin{bmatrix}a+b & b+c \\ c+d & d + a \end{bmatrix}$.  What is the $\ker(\Phi)$? 
<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
he matrix $\bM = \begin{bmatrix} 0 & 1 \\ -1 & 0 \end{bmatrix}$ rotates a vector clockwise 90 degrees.  Determine the matrix that rotates a vector 90 degrees clockwise in the basis $\bB = \set{\begin{bmatrix}1\\3 \end{bmatrix}, \begin{bmatrix}2\\-1 \end{bmatrix}}$.
<div style="text-align: right;">
[Solution]( )
</div> 
</div>



<div class="exercise">
Suppose Alice has standard basis vectors $e_1$ and $e_2$.  Let Bob have basis vectors given by $\bb_1 = \begin{bmatrix}1\\2 \end{bmatrix}$ and $\bb_2 = \begin{bmatrix} -1\\1 \end{bmatrix}$.
           \begin{enumerate}
            \item   Take vector $\bv_b = \begin{bmatrix} 3\\5 \end{bmatrix}$ and map it into Alice's world. \vfill
            \item   What are Alice's basis vectors in terms of Bob's basis? \vfill
            \item   Perform a 90 degree rotation in Alice's world for the vector $\begin{bmatrix} 4\\ 1 \end{bmatrix}$ in Bob's world and move it back to Bob's world.
<div style="text-align: right;">
[Solution]( )
</div> 
</div>




<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->
<!-- NEW SECTION --><!-- NEW SECTION --><!-- NEW SECTION -->






## Affine Spaces

Affine spaces are geometric spaces that are offset from the origin. Unlike vector subspaces, they do not necessarily contain the zero vector.  Mappings between affine spaces share many properties with linear mappings.


<div class="note">
In the machine learning literature, the terms *linear* and *affine* are often used interchangeably.
</div>



<div class="definition"> 
Let \( V \) be a vector space, \( x_0 \in V \), and \( U \subseteq V \) a subspace.  Then
\[
L = x_0 + U := \{ x_0 + u : u \in U \}
\]
is called an **affine subspace** or **linear manifold** of \( V \).

- \( U \) is the **direction space**.  
- \( \mathbf{x}_0 \) is the **support point**.  
- If \( \mathbf{x}_0 \notin U \), then \( L \) is *not* a linear subspace (it does not contain the origin).
</div>



<div class="example">
Points, lines, and planes in \( \mathbb{R}^3 \) that do not necessarily pass through the origin are examples of affine spaces.  
</div>


<div class="definition">  
If \( L = \mathbf{x}_0 + U \) and \( (\mathbf{b}_1, \ldots, \mathbf{b}_k) \) is a basis of \( U \),  
then every \( \mathbf{x} \in L \) can be written as:
\[
\mathbf{x} = \mathbf{x}_0 + \lambda_1 \mathbf{b}_1 + \cdots + \lambda_k \mathbf{b}_k, \quad \lambda_i \in \mathbb{R}
\]
This is called the **parametric equation** of the affine subspace.
</div>



<div class="example">
-   A line \( y = x_0 + \lambda b_1 \), where \( \lambda \in \mathbb{R} \)  is an example of a parametric equation in a 1-dimensional affine subspace.

- A plane \( y = \mathbf{x}_0 + \lambda_1 \mathbf{b}_1 + \lambda_2 \mathbf{b}_2 \),  where \( \mathbf{b}_1, \mathbf{b}_2 \) are linearly independent  is an example of a parametric equation in a 2-dimensional affine subspace.

- A hyperplane is an example of an \((n - 1)\)-dimensional affine subspace
  \[
  y = \mathbf{x}_0 + \sum_{i=1}^{n-1} \lambda_i \mathbf{b}_i
  \]
  - In \( \mathbb{R}^2 \), a line is a hyperplane.  
  - In \( \mathbb{R}^3 \), a plane is a hyperplane.
</div>


---

### Relation to Linear Equations

For \( \mathbf{A} \in \mathbb{R}^{m \times n} \) and \( \mathbf{x} \in \mathbb{R}^m \), the solution set of \( \mathbf{A}\lambda = \mathbf{x} \) is either empty or an **affine subspace** of \( \mathbb{R}^n \)  with dimension \( n - \text{rk}(\mathbf{A}) \).

- The equation \( \mathbf{A} \mathbf{x} = \mathbf{b} \) (inhomogeneous system) defines an affine subspace.  
- The equation \( \mathbf{A} \mathbf{x} = \mathbf{0} \) (homogeneous system) defines a vector subspace,  which can be seen as a special affine subspace with support point \( \mathbf{x}_0 = \mathbf{0} \).

---

### Affine Mappings

Affine mappings generalize linear mappings by including a **translation**.



<div class="definition">  
Let \( V, W \) be vector spaces, \( \Phi : V \to W \) a linear map, and \( \mathbf{a} \in W \).  Then
\[
\varphi : V \to W, \quad x \mapsto \mathbf{a} + \Phi(\mathbf{x})
\]
is an **affine mapping** with **translation vector** \( \mathbf{a} \).
</div>



<div class="lemma">
Properties:

- Every affine map can be written as a composition:
  \[
  \varphi = \tau \circ \Phi
  \]
  where \( \tau \) is a translation and \( \Phi \) is linear.  
- The composition of affine mappings is **affine**.  
- If an affine mapping is bijective, it preserves:
  - **Dimension**  
  - **Parallelism**  
  - **Geometric structure**  
  </div>


---



### Exercises {.unnumbered .unlisted}


Put some exercises here.