Information
AI Chat

1.3 Linear Algebra - junior

junior

Course

english 101 (Eng101)

14 Documents

Students shared 14 documents in this course

University

University of Shendi

Academic year: 2018/2019

Uploaded by:

Ali Abdul-Wahab

University of Shendi

0followers

9Uploads

1upvotes

Comments

Please sign in or register to post comments.

Preview text

Linear Algebra for Machine

Learning

Sargur N. Srihari

srihari@cedar.buffalo

What is linear algebra?

• / Linear algebra is the branch of mathematics

concerning linear equations such as

a

1 x

1 +.....+ a

n

x

n

= b

– /In vector notation we say a

T

x = b

– /Called a linear transformation of x

• / Linear algebra is fundamental to geometry, for

defining objects such as lines, planes, rotations

Linear equation a 1 x 1 +.....+ a n x n= b

defines a plane in ( x 1 ,.., x n) space

Straight lines define common solutions

to equations

Linear Algebra Topics

– /Scalars, Vectors, Matrices and Tensors

– /Multiplying Matrices and Vectors

– /Identity and Inverse Matrices

– /Linear Dependence and Span

– /Norms

– /Special kinds of matrices and vectors

– /Eigendecomposition

– /Singular value decomposition

– /The Moore Penrose pseudoinverse

– /The trace operator

– /The determinant

– /Ex: principal components analysis

####### 4

Scalar

• / Single number

– /In contrast to other objects in linear algebra,

which are usually arrays of numbers

• / Represented in lower-case italic x

– /They can be real-valued or be integers

• / E., let be the slope of the line

/Defining a real-valued scalar

• / E., let be the number of units

/Defining a natural number scalar

x *!

n *!

Matrices

• / 2 -D array of numbers

– /So each element identified by two indices

• / Denoted by bold typeface A

– /Elements indicated by name in italic but not bold

• / A

1,

is the top left entry and A

m,n

is the bottom right entry

• / We can identify nos in vertical column j by writing : for the

horizontal coordinate

• / E.,

• / A

i :

is i

th

row of A , A

:j

is j

th

column of A

• / If A has shape of height m and width n with

real-values then

A =

A

£

§

A *!

####### m × n

Tensor

• / Sometimes need an array with more than two

axes

– /E., an RGB color image has three axes

• / A tensor is an array of numbers arranged on a

regular grid with variable number of axes

– /See figure next

• / Denote a tensor with this bold typeface: A

• / Element ( i,j,k ) of tensor denoted by A

i,j,k

Transpose of a Matrix

• / An important operation on matrices

• / The transpose of a matrix A is denoted as A

T

• / Defined as

( A

T

)

i,j

= A

j,i

– /The mirror image across a diagonal line

• / Called the main diagonal , running down to the right

starting from upper left corner

A =

A 1,1 A 1,2 A 1,

A 2,1 A 2,2 A 2,

A 3,1 A 3,2 A 3,

£ £ £ £

§ § § §

ó A T =

A 1,1 A 2,1 A 3,

A 1,2 A 2,2 A 3,

A 1,3 A 2,3 A 3,

£ £ £ £

§ § § §

A =

A 1,1 A 1,

A 2,1 A 2,

A 3,1 A 3,

£ £ £ £

§ § § §

ó A T =

A 1,1 A 2,1 A 3,

A 1,2 A 2,2 A 3,

£ £ £ £

§ § § §

Vectors as special case of matrix

• / Vectors are matrices with a single column

• / Often written in-line using transpose

x = [ x

1 ,..,x

n

]

T

• / A scalar is a matrix with one element

a=a

T

x =

x 1

x 2

£ £ £ £ £ £ £ £

§ § § § § § § §

ó x T =££ x 1 ,x 2 ,. §§

Multiplying Matrices

• / For product C = AB to be defined, A has to have

the same no. of columns as the no. of rows of B

– /If A is of shape m x n and B is of shape n x p then

matrix product C is of shape m x p

– /Note that the standard product of two matrices is

not just the product of two individual elements

• / Such a product does exist and is called the element-wise

product or the Hadamard product A ¤ B

C = AB ó C

i,j

= A

i,k k

3 Bk,j

Multiplying Vectors

• / Dot product between two vectors x and y of

same dimensionality is the matrix product x

T

y

• / We can think of matrix product C=AB as

computing C

ij

the dot product of row i of A and

column j of B

Example flow of tensors in ML

A linear classifier y = Wx T+ b

A linear classifier with bias eliminated y = Wx T

Vector x is converted

into vector y by

multiplying x by a matrix W

Linear Transformation

• / A x = b

– /where and

– /More explicitly

• / Sometimes we wish to solve for the unknowns

x ={ x

1 ,..,x

n

} when A and b provide constraints

A *!

####### n × n

b *!

####### n

A 11 x 1 + A 12 x 2 + ....+ A1nxn= b 1

A 2 1 x 1 + A 2 2 x 2 + ....+ A 2 nxn= b 2

A

n 1

x

+ A

m 2

x

+ ....+ A

n , n

x

= b

n equations in

n unknowns

A =

A 1,1! A 1, n

""" An ,1! Ann

£ £ £ £

§ § § §

x =

x 1

" xn

£ £ £ £

§ § § §

b =

b 1

" bn

£ £ £ £

§ § § §

n x n n x 1 n x 1

Can view A as a linear transformation

of vector x to vector b

Matrix Inverse

• / Inverse of square matrix A defined as

• / We can now solve A x = b as follows:

• / This depends on being able to find A

- 1

• / If A

- 1

exists there are several methods for

finding it

A

####### 21

A = I

####### n

A x = b

A

A x = A

b

I

x = A

b

x = A

b

Solving Simultaneous equations

• / Ax = b

where A is ( M +1) x ( M +1)

x is ( M +1) x 1 : set of weights to be determined

b is N x 1

Was this document helpful?