**EE 527: Detection and Estimation Theory
(Spring 2016)**

**Updates/Reminders****Week 1 and 2: Recap and new background needed in probability and linear algebra****Prerequisites:**EE 224, EE 322, Basic calculus & linear algebra. Suggested co-requisite: EE 523**Location, Time:****Coover 3138, Tues-Thurs 2:10-3:30****Instructor:**Prof Namrata Vaswani**Office Hours: Monday 10-11, Wednesday 2-3****Office:**3121 Coover Hall**Email:**namrata AT iastate DOT edu**Phone:**515-294-4012**Grading policy****Homeworks: 10%****Two midterms and one final exam : 20%, 20%, 40%****One project / term paper: 10%****Exam Dates and Project Details and Deadlines****Exam dates****Midterm 1:****Midterm 2**:**Project details:****TBD****Syllabus:**- Background material:
recap of probability, calculus, linear algebra
- Estimation Theory
- Minimum variance
unbiased estimation, best linear unbiased estimation
- Cramer-Rao lower bound
(CRLB)
- Maximum Likelihood
estimation (MLE): exact and approximate methods (EM, alternating max,
etc)
- Bayesian inference
& Least Squares Estimation (from Kailath et al's Linear
Estimation book)
- Basic ideas, adaptive
techniques, Recursive LS, etc
- Kalman filtering
(sequential Bayes)
- Finite state Hidden
Markov Models: forward-backward algorithm, Viterbi (ML state
estimation), parameter estimation (f-b + EM)
- Graphical Models
**Applications**: image processing, speech, communications (to be discussed with each topic)- Sparse Recovery and
Compressive Sensing introduction
- Monte Carlo methods:
importance sampling, MCMC, particle filtering, applications in numerical
integration (MMSE estimation or error probability computation) and in
numerical optimization (e.g. annealing)
- Detection Theory
- Likelihood Ratio
testing, Bayes detectors,
- Minimax detectors,
- Multiple hypothesis
tests
- Neyman-Pearson
detectors (matched filter, estimator-correlator etc),
- Wald sequential test,
- Generalized likelihood
ratio tests (GLRTs), Wald and Rao scoring tests,
- Applications
- The syllabus is similar to Prof.
Dogandzic's EE527
but I will cover least squares estimation, Kalman filtering and Monte
Carlo methods in more detail and will discuss some image/video processing
applications also. Note that LSE, KF are also covered in EE524, but
different perspectives are always useful
**Books:****Textbook: S.M. Kay's Fundamentals of Statistical Signal Processing: Estimation Theory (Vol 1), Detection Theory (Vol 2)****References**- Kailath, Sayed and
Hassibi,
*Linear Estimation* - V. Poor,
*An Introduction to Signal Detection and Estimation* - H.Van Trees,
*Detection, Estimation, and Modulation Theory* - J.S. Liu,
*Monte Carlo Strategies in Scientific Computing.*Springer-Verlag, 2001. - B.D. Ripley,
*Stochastic Simulation.*Wiley, 1987.

**Disability accommodation:**If you have a documented disability and anticipate needing accommodations in**Homeworks****Homework 1: due Tues Jan 26**- New added: Prove all the if and only if statements for
joint-Gaussian random variables from this
document
- Chapter 3 of Supplementary
Problems for Bertsekas’s Probability Text: Problems 5, 6, 8, 9, 10, 14, 18, 19,
20, 21
**Correction:****Suppose X1 is N(0,1), X2 is 1 w.p. 1/2 and -1 w.p. 1/2, and X3 = X1. X2. Compute the pdf of X3 and compute the joint pdf of X1 and X3**- Practice problems: use this link to do selected practice problems
from Chapters 1 and 2: EE 322 Fall 2007 homework sets (do not need to be submit)
**Homework 2: due Tues Feb 16**- Problems 2.1, 2.4, 2.7, 2.9, 2.10 of Kay-I. Bonus: 2.8
**Homework 3: due Tues Feb 23**- Problems 5.2, 5.3,
5.4, 5.5, 5.7, 5.13, 5.16
- Compute
the MVUE for a N(\mu,\sigma^2) distribution using N i.i.d. observations.
Also compute the covariance matrix of the MVUE estimator
**Course Handouts****Introduction slides****Linear Algebra and Probability Review and New Material**- Basic probability review (EE 322 recap)
- Practice problems: http://athenasc.com/prob-supp.html
(look at problems for Chapter 1, 2, 3).
- EE
322 course
**Probability Review - 2**(Prof. ALD's notes)- New Probability Background Needed
- Linear Algebra Review and New Material
**Classical Estimation****Recap notes for all topics: fill in the gaps in the notes below****min (classical) MSE estimation, MVUE, Sufficient Statistics, MLE**- H1: Minimum Variance Unbiased
Estimation (Prof. ALD's notes), Chapter 2, 3, 5 of Kay-I
**H2: Cramer Rao Bound and Efficient Estimators**(Prof. ALD's notes)- H3: Linear Models, Best Linear
Unbiased Estimator and ML Estimation (Prof. ALD's notes)
**EM algorithm**- Dempster paper,
- See scanned notes in
WebCT
- Prof. ALD's handout
**Least Squares estimation****LS estimation summary**- Also see Chapter 8
of Kay-I
**Sparse Recovery / Compressive sensing****Introduction****Old slides: Introduction to Compressive Sensing**- Compressive
Sensing class (EE 520)
- CS papers' archive
**Bayesian estimation****MMSE and linear MMSE estimation and Kalman filtering**- Joint
Gaussian random variables, MMSE and linear MMSE Estimation: based
on Poor’s book
- Kalman
filtering details and proofs: based on Poor's book
**Kalman Filter algorithm summary (an old handout)**: based on Poor's book- See Spring 2008
midterm 2 solutions (in WebCT) for
- second formula for
Kalman gain
- KF with control
input
- Extended KF
- KF for case when
system noise and observation noise at the same time are correlated
**Some extra things****Bayesian Inference using Prof ALD's notes**: some parts of this are useful**Graphical models****Graphical models**(Prof. ALD's notes) approach for handling conditional dependencies in Bayesian estimation (Prof ALD's handout)**Hidden Markov Models (HMM)**- HMM
notes: mostly based on Rabiner's tutorial paper
**(new notes)** - Rabiner's tutorial paper
**Detection Theory:**- H5, H5b ,
**H5c**(Prof ALD's notes) - Generalized
LRT: asymptotic distribution
**Monte Carlo****Simple MC and Importance Sampling (IS)**- Simple MC, Importance Sampling (IS),
Biased IS & Bayesian IS
**(Prof ALD's handout)** **Geweke's paper (1989)**- Concept, Conditions
for getting Consistent Estimates
- Bias-variance
tradeoff, Finding good importance densities (to reduce estimator
variance), Examples
- Resampling
(sampling-importance resampling), Rao-Blackwellization
- Techniques for
generating random samples from a PDF
- Accept-Reject
**Markov Chain Monte Carlc (MCMC)****Markov chain concepts and Markov Chain Monte Carlo (Prof ALD's handout)**- Markov chain
concepts (for proofs, full details, take EE 523)
- General conditions
for MCMC to work (slide 24)
- Metropolis-Hastings
and proof of why it works
**Particle filtering**- Particle filtering
**Doucet et al's paper (2000)**- HMM model and other
algorithms
- Importance sampling
to approximate a PDF (sum of weighted Diracs)
- Sequential Importance
Sampling (SIS)
- Resampling concept
- Particle filtering
algorithm: SIS + Resample
**Probability and Linear Algebra Recap****Undergrad probability review (EE 322)**- Chapter
1 slides
**Single Random Variable: Discrete & Continuous****Multiple Random Variables: Discrete**, Multiple Random Variables: Continuous**Linear algebra review:**- http://www.maths.mq.edu.au/~wchen/lnlafolder/lnla.html
(Chapters 5-10, Chap 11,12 may also be useful)