CS598 - Robot Perception

Date	Lecture	Readings	Logistics
Module 1: Introduction and Background
8/25	Lecture #1 (Shenlong): Introduction to Robot Perception [ slides \| video \| notes ]	Sünderhauf et al., The Limits and Potentials of Deep Learning for Robotics Lake et al., Building Machines That Learn and Think Like People
8/27	Lecture #2 (Shenlong): Poses, Transforms and Kinematics - 3D Transformations - Rotation Representations - Kinematics [ slides \| video \| notes ]	Probabilistic Robotics, Ch. 5 State Estimation on Robotics, Sec. 6.1 - Sec. 6.3 Sola et al., A micro Lie theoryfor state estimation in robotics
9/1	No class
9/3	Lecture #3 (Shenlong): Sensing I - LiDAR, Radar, Sonar - GPS, IMU, Odometer - Touch, Tactile, etc [ slides \| video \| notes ]	State Estimation on Robotics, Sec. 6.4 Introduction to Autonomous Mobile Robots, Sec. 4.1	Presentation Preference due (11:59pm, Fri 8/27)
9/8	Lecture #4 (Shenlong): Sensing II - Cameras - Image Formation - Perspective Geometry [ slides \| video \| notes ]	State Estimation on Robotics, Sec. 6.4 Computer Vision, Chapter 2, 12
9/10	Lecture #5 (Shenlong): Sensing III - Stereo, Structured Light and Event Cameras - Epipolar Geometry - Structure-from Motion [ slides \| video \| notes ]	State Estimation on Robotics, Sec. 6.4 Computer Vision, Chapter 2, 12
9/15	Lecture #6 (Shenlong): Deep Learning Review - MLP, Backprop - CNNs, RNNs, Transformers, GNNs - Generative Models [ slides \| video \| notes ]	Deep Learning, Chap. 6, 9, 10, 20 CS498 Intro to DL Relavent Slides [Demo] Pytorch Tutorials Learning Basics
9/17	Lecture #7 (Shenlong): 3D Representations - Voxel, Mesh, Points, SDFs - Representation Learning [ slides \| video \| notes ]	Tutorial on 3D Deep Learning Slides [R] Park et al., DeepSDF Learning Continuous Signed Distance Functions for Shape Representation [R] Hornung et al., OctMap An Efficient Probabilistic 3D Mapping Framework Based on Octrees [Demo] Open3D, Point Cloud, Mesh, RGBD Images	Paper reviewing starts from this lecture. Choose from papers starting with [R] to review.
9/22	Lecture #8 (Shenlong): Graphical Models and Nonlinear Optimization Review - MRFs, Bayes Nets and Factor Graphs - Map Inference, Marginal Inference - Nonliner Optimization [ slides \| video \| notes ]	Factor Graphics for Robot Perception, Chapter 1, 2 [R] Krahenbuhl et al., Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials [R] Davison et al., FutureMapping 2 - Gaussian Belief Propagation for Spatial AI [Demo] Ortiz et al., A visual introduction to Gaussian Belief Propagation	Presentation assignments out (11:59pm, Fri 9/17)
Module 2: Case Studies
9/24	Lecture #9 (Shenlong): Spatial Perception: Map-based Localization - State Estimation Review - Map Representations - Bayes Filtering, Kalman Filters - Particle Filters, Histogram Filters - Registration and Matching [ slides \| video \| notes ]	Probabilistic Robotics, Ch. 9 Probabilistic Robotics, Chap 2, 3, 4 [R] Delleart et al. Monte Carlo Localization for mobile robots, [R] Bloesch et al. State Estimation for Legged Robots - Consistent Fusion of Leg Kinematics and IMU [Demo] Labee et al. Kalman and Bayesian Filters in Python [R] Levinson et al., Robust vehicle localization in urban environments using probabilistic maps [R] Jegou et al., Aggregating local descriptors into a compact image representation Barsan et al., Learning to Localize Using a LiDAR Intensity Map	Project Proposal due (11:59pm, Fri 9/24)
9/29	Lecture #10 (Shenlong): Spatial Perception: SLAM - RGBD and LiDAR SLAM [ slides \| video \| notes ]	[R] Newcombe et al., KinectFusion Real-Time Dense Surface Mapping and Tracking [R] Grisetti et al., Improved Techniques for Grid Mapping with Rao-Blackwellized Particle Filters Cadena et al., Past, Present, and Future of Simultaneous Localization And Mapping
10/1	Lecture #11 (Issac Ngui and Emersion Sie): Spatial Perception: SLAM - Visual Odometry - Indirect SLAM - Direct SLAM [ slides \| video \| notes ]	[R] Newcombe et al., DTAM Dense Tracking and Mapping in Real-Time [R] Kitt et al., Visual odometry based on stereo image sequences with RANSAC-based outlier rejection scheme [R] Mur-Artal et al., ORB-SLAM2 an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras Cadena et al., Past, Present, and Future of Simultaneous Localization And Mapping Sarlin et al., From Coarse to Fine Robust Hierarchical Localization at Large Scale
10/6	Lecture #12 (Abid and Jeffrey Zhang): Object Perception: Detection and Tracking - 2D and 3D Detection - 2D and 3D Tracking [ slides \| video \| notes ]	[R] He et al., Mask r-cnn [R] Redmon et al., YOLO9000 better, faster, stronger [R] Yang et al., Pixor Real-time 3d object detection from point clouds [R] Held et al., Learning to Track at 100 FPS with Deep Regression Networks Weng et al., 3D Multi-Object Tracking A Baseline and New Evaluation Metrics [Demo] Pytorch Tutorials Object Detection
10/8	Lecture #13 (Zhijian Liu): Efficient 3D Perception [ slides \| video \| notes ]
10/13	Lecture #14 (Shenlong Wang): Spatial Perception: Semantic Scene Understanding - 2D Semantic Segmentation - Instance and Panoptic Segmentation [ slides \| video \| notes ]	[R] Ronneberger et al., U-Net Convolutional Networks for Biomedical Image Segmentation [R] Zhao et al., Pyramid Scene Parsing Network [R] Krillov et al. Panoptic Segmentation
10/15	Lecture #15 (Joao Marcos Correia Marques, Shreya Gummadi): Spatial Perception: Semantic Scene Understanding - 3D Segmentation - Semantic Fusion [ slides \| video \| notes ]	[R] Qi et al., PointNet++ Deep Hierarchical Feature Learning on Point Sets in a Metric Space [R] Wang et al., Deep parametric continuous convolutional neural networks [R] McCormac et al., SemanticFusion Dense 3D Semantic Mapping with Convolutional Neural Networks
10/20	Lecture #16 (Mengchao Zhang, James Motes): Object Perception: Pose Estimation - 6-DoF Pose Estimation - Articulated Pose Estimation [ slides \| video \| notes ]	[R] Xiang et al., PoseCNN A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. [R] Wang et al., DenseFusion 6D Object Pose Estimation by Iterative Dense Fusion. [R] Cao et al., OpenPose Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
Module 3: Challenges and Frontiers
10/22	Lecture #17 (Yifan Zhao, Yixuan Wang): Perception in a Moving World - Optical Flow - Nonrigid Tracking [ slides \| video \| notes ]	[R] Baker et al., Lucas-Kanade 20 Years On A Unifying Framework. [R] Teed et al., RAFT Recurrent All-Pairs Field Transforms for Optical Flow. [R] Newcombe et al., DynamicFusion Reconstruction and Tracking of Non-rigid Scenes in Real-Time.
10/27	Lecture #18 (Sayantani Basu, Jiachen Li): Perception in a Moving World - Motion Forecasting - Streaming Perception [ slides \| video \| notes ]	[R] Kitani et al., Activity Forecasting. [R] Alahi et al., Social LSTM Human Trajectory Prediction in Crowded Spaces [R] Li et al., Towards Streaming Perception. Liang et al., Learning Lane Graph Representations for Motion Forecasting Cao et al., Long-term Human Motion Prediction with Scene Context Teed et al., Human Motion Trajectory Prediction A Survey
10/29	Lecture #19 (Shenlong Wang, Roger Qiu): Perception in an Open World - Open-set Perception - Out-of-Distribution Robustness - Continual / Life-long Learning - Exploration [ slides \| video \| notes ]	[R] Bendale et al., Towards Open World Recognition [R] Wong et al., Identifying Unknown Instances for Autonomous Driving [R] Li et al., Learning without Forgetting [R] Jayaraman et al., Learning to Look Around Intelligently Exploring Unseen Environments for Unknown Tasks. Dinesh Jayaraman, Kristen Grauman (2017)	Mid-term Presentaion (4:15pm, Fri 10/29)
TBD	Lecture #20 : Mid-term Project Presentations [ slides \| video \| notes ]
11/3	Lecture #21 (Mohit Goyal, Sara Aghajanzadeh): Perception and Action - Active Perception [ slides \| video \| notes ]	[R] Bohg et al., Interactive Perception Leveraging Action in Perception and Perception in Action [R] Pinto et al., The Curious Robot Learning Visual Representations via Physical Interactions
11/5	Lecture #22 (Shaowei Liu, Aditya Prakash): Perception and Action - Perceive for manipulation - Perceive for navigation [ slides \| video \| notes ]	[R] Zhou et al., Does computer vision matter for action? [R] Lenz et al., Deep learning for detecting robotic grasps [R] Chaplot et al., Neural Topological SLAM for Visual Navigation Saxena et al., Robotic Grasping ofNovel Objects using Vision Mahler et al., Dex-Net 2.0 Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics Bertinetto et al., Dense Object Nets Learning Dense Visual Object Descriptors By and For Robotic Manipulation Philion et al., Learning to Evaluate Perception Models Using Planner-Centric Metrics
11/10	Lecture #23 (German Ros): Invited Speaker Talk: TBD [ slides \| video \| notes ]
11/12	Lecture #24 (Kaiwen Hong, Kendall Koe): Multi-Modal Perception - Multi-sensor Perception [ slides \| video \| notes ]	[R] Liang et al., Deep continuous fusion for multi-sensor 3d object detection [R] Chen et al., SoundSpaces Audio-Visual Navigation in 3D Environments [R] Lee et al., Making Sense of Vision and Touch Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks Leutenegger et al., Keyframe-based visual–inertial odometry using nonlinear optimization
11/17	Lecture #25 (Shenlong Wang, Arjun Gupta): Perception and Simulation - Intro to Simulation - Sim2Real Domain Transfer [ slides \| video \| notes ]	[R] Richter et al., Playing for Data Ground Truth from Computer Games [R] Hoffman et al., CyCADA Cycle-Consistent Adversarial Domain Adaptation Richter et al., Enhancing Photorealism Enhancement [Sim Env] Carla, AirSim, AI-Habitat, iGibson, etc. [Sim Data] TartanAir, Sintel, Replica, etc.
11/19	Lecture #26 (Yuan Shen, Vlas Zyrianov): Learning to Sim - Realistic Sensor Simulation - Scenario Simulation [ slides \| video \| notes ]	[R] Riegler et al., Stable View Synthesis [R] Manivasagam et al., LiDARsim Realistic LiDAR Simulation by Leveraging the Real World [R] Devaranjan et al., Meta-Sim2 Unsupervised Learning of Scene Structure for Synthetic Data Generation Mildenhall et al., NeRF Representing Scenes as Neural Radiance Fields for View Synthesis Dellert et al., NeRF Explosion 2020 Wang et al., Paired Open-Ended Trailblazer (POET) Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
11/24	No class. Thanksgiving
11/26	No class. Thanksgiving
12/1	Lecture #27 (Luca Carlone): Invited Speaker Talk: TBD [ slides \| video \| notes ]
12/3	Lecture #28 (Amish Goel, Patrick Naughton): Perception with Guarantees - Uncertainty Modeling - Adversarial Robustness - Certifiable Perception [ slides \| video \| notes ]	[R] Guo et al., On calibration of modern neural networks [R] Goodfellow et al., Explaining and Harnessing Adversarial Examples [R] Eykholt et al., Robust physical-world attacks on deep learning visual classification Cohen et al., Certified Adversarial Robustness via Randomized Smoothing Yang et al., TEASER Fast and Certifiable Point Cloud Registration Kendall et al., What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision Jonschkowski et al., Differentiable Particle Filters End-to-End Learning with Algorithmic Priors Lakshminarayanan et al., Introduction to Uncertainty in Deep Learning	Extended office hour for final project (4:15pm, Fri 12/3)
Module 4: Final Project Presentations
12/8	Final presentations (2pm to 4pm)