Perception
From Princeton Autonomous Vehicle Engineering
In the general robotics paradigm of PCASE, perception is concerned with obtaining information about robot's environment and current state, through the use of various sensors. the PAVE perception team is a software team which is responsible for the implementation of the various software software modules required to acquire and process sensor data and interface with Cognition.
Contents |
Overview
The overall goal of perception, and the perception team, is to build a cohesive representation of the environment and the vehicle state. The environment is represented by the boundaries which the vehicle must operate within, and the obstacles, both moving and stationary, that are within and near those boundaries. The state is produced by state estimation, which tracks estimates of position, heading, velocity and other similar parameters over time. Perception is responsible for acquiring and processing the data from the various state sensors for State Estimation to use.
The bulk of the problems in perception are in the sensing and processing of data obtained from the environment around the vehicle. Although there are many types of environmental sensors, outlined below, at PAVE we primarily use cameras to detect both obstacles and lanes, and we supplement this information with additional support from RADAR and ultrasonics. In addition there are many more problems that need to be dealt with in terms of image data, such as detection of trafic lights and the state of the light, various road signs, recognition of cars and people, etc.
Environment Sensing
Sensors such as RADAR, Ultrasonic, Stereo Cameras, Monocular Cameras, and LIDAR provide robots with the capability to sense objects and boundaries within their environment. Environment sensors are divided into two categories: active and passive. Passive sensors rely on detecting energy already present in the environment, such as light reflected off objects or heat radiating off objects. Cameras are classified as passive sensors. RADAR, ultrasonic, and LIDAR are all examples of active sensors. Active sensors send energy into the environment and measure the reflection of that energy off surfaces.
In order to find obstacles, the information returned by both classes of sensors must be post-processed by various obstacle detection algorithms. The final perception step is to fuse the various sources of obstacle data together to create a complete representation of each discrete object using data gathered through the various sources. This is accomplished through obstacle Fusion.
In order to detect lanes, LIDAR may be used to detect the thickness or reflectivity of the paint used. However, passive sensors such as cameras are much more reliable. through lane detection, the image is processed to find lanes, which are then tracked in lane tracking.
State Sensing
A variety of sensors such as GPS, IMU/accelerometers, encoders, and digital compasses are employed to sense the vehicle's speed, heading, position, and other parameters.
- GPS provides a relatively accurate indication of global position, and can provide estimates of speed and heading for a moving vehicle.
- IMU or accelerometers measure the acceleration of a vehicle along various axes. This can be used to measure how fast a vehicle is turning, and can also be used to figure out the dynamics of a vehicle.
- encoders can measure a rotary position, either absolute or relative. This allows the measurement of speeds of one wheel of a vehicle. Individual measurements can be combined to yield differential speed measurements. (We might not be using this because it is worse than an IMU measurement while providing little new information)
- A digital compass is used to give an accurate measurement of the vehicle's heading, even when the vehicle is at a standstill. We won't be using one because it is too easily biased by disturbances in the car.
The data from all of these sensors is sent to State Estimation, which aggregates and filters the various data sources to produce one coherent state vector with a good estimate of vehicle speed, position, heading, and other parameters.
Recent News
The solution where we'll be implementing the Camera API is in PAVE_Common/Libraries. You may not have the PAVE_Common repository checked out yet; check Source Control for instructions. Look at Camera.h and Frame.h to learn the API we'll be transitioning to, then at CameraLive.cpp and CameraFiles.cpp for a taste of what was done in the past. Camera API people, you can copy over code from those two .cpp files, but make new subclasses like VidereCamera and FlycaptureCamera (don't need to use those names) for the new implementations. Need to remove the old .cpp files before we can build. People looking at integrating obs. and lane detection will need to look at the APIs too. And IPC people, look at the IGVC code for what's been done, although feel free to redesign if it will be more efficient. As for the APIs, they're not perfect, but we need somewhere to start, modify them if you have to but I think that the general spirit of how they work is good. I tried to allow us to reuse as much code both on the camera backend and the image processing frontend as possible, including minimal changes to this summer's OD code. No impedance mismatches AFAIK. Brenton 18:36, 13 November 2009 (EST)
Sub-Projects
There are a variety of Perception sub-projects, each summarized below. Visit the main page for each project to see more specific references, progress, and documentation for each project. For the most part, there are working versions of all of the different projects listed below. However, there is always room for improvement.
Obstacle Detection
The main page for stereo obstacle detection is Obstacle Detection. Information for RADAR can be found on the RADAR page.
This is the element of perception that sits at the interface of software and hardware. Obstacle Detection itself is actually a collection of sub-projects, all with the aim of extracting data about obstacles from a sensor and presenting it in a neat, well-packaged form to Obstacle Fusion. Most of the interesting problems in this category fall into the interpretation of obstacle data. For example, it is important to find a fast and efficient way of taking a 3-D point cloud and finding the individual obstacles jutting out from the ground plane and representing them in a way that is easily manipulated but neatly encapsulates the size and characteristics of the object mapped by the point cloud.
Obstacle Fusion and Tracking
Main Article: Obstacle Fusion
In the case of the IGVC vehicles, there is only one environmental sensor, so no fusion is necessary. However, with a complex system such as an autonomous car, there are multiple sensors, each with different detection range and field of view, sometimes with overlapping detections and information, and sometimes with different types of information. It is up to Obstacle Fusion to take the various sets of obstacles detected by the obstacle detection modules and match and combine the detections together to form one good estimate of an obstacle which can then be tracked.
Some of the problems in obstacle matching include: how to accurately match up the sensor data from multiple sources, how to represent the obstacles so that they can be easily tracked through time, and how to match obstacles detected in subsequent frames to ones detected previously, taking occlusion and uncertain motion of the obstacle into account.
Lane Detection
Main Article: Lane Detection
The problem of lane detection is to build a system capable of detecting line boundaries while being robust to various environmental conditions. Sometimes, the limitation of the camera becomes problematic, since the camera is unable to evenly expose scenes where there are shadows, especially when the camera is in the shade peering out into bright areas. In addition, dappled shadows can fool edge detection algorithms into marking the bright areas caused by the dappled shadow as edges, increasing the possibility of false-positive lane detection. Although there has been a lane detection algorithm under development for some time, the problem of robustness to challenging lighting conditions still remains.
Lane Fusion/Tracking
Main Article: Lane Tracking
Since all lane detection algorithms are prone to missed detections and false positives, it is up to the Lane Fusion algorithm to remove false positives and compensate for missed detections by providing an estimate of the lanes ahead according to successful detections from previous frames. Some of the problems inherent in Lane Fusion are: how evaluate lanes to detect and remove false positives, how to increment the lane estimate forward in time, and what lane parameters are useful in estimating and tracking lanes through time.
Environmental Sensing
- GPS
- IMU/Gyroscope/Accelerometer
- Compass
- Encoders
Additional Projects of Interest
These are additional projects that are not on the critical timeline for any PAVE projects at the moment but are open for exploration for curious engineers/programmers:
- Road Detection - This project is actually a superset of Lane Detection, involving the detection of a road, or flat drivable surface in the absence of lanes.
- Car Detection - Given an image, tell whether there is a car in the image and where it is.
- Object Recognition - A general algorithm that can be trained to recognize various objects.
- Traffic Light Recognition - Find traffic lights in an image and recognize whether the light is green, yellow, or red.
- Sign Reading Recognize various standard road signs in images, also possibly read street signs.
