Simulators for Autonomous Vehicles, Perception Review, Spain, 2019

September 3, 2019, 12:45 PM

in Sensors... group at U Cartagena  (Spain).

Rosique, Navarro, Fernandez and Padilla

in Div of EE.

Abstract: This paper presents a systematic review of the perception systems and simulators for autonomous vehicles (AV). This work has been divided into three parts. In the first part, perception systems are categorized as environment perception systems and positioning estimation systems. The paper presents the physical fundamentals, principle functioning, and electromagnetic spectrum used to operate the most common sensors used in perception systems (ultrasonic, RADAR, LiDAR, cameras, IMU, GNSS, RTK, etc.). Furthermore, their strengths and weaknesses are shown, and the quantification of their features using spider charts will allow proper selection of different sensors depending on 11 features. In the second part, the main elements to be taken into account in the simulation of a perception system of an AV are presented. For this purpose, the paper describes simulators for model-based development, the main game engines that can be used for simulation, simulators from the robotics field, and lastly simulators used specifically for AV. Finally, the current state of regulations that are being applied in different countries around the world on issues concerning the implementation of autonomous vehicles is presented.

1. Introduction
Rapid advances in electronics, information, and communications technology (leading to miniaturization and improvement of computers, sensors and networking performance) have given
rise to the development of several autonomous vehicles (AV) technologies [1]. UCSUSA [2] defines autonomous vehicles as follows: “Self-driving vehicles as cars or trucks in which human drivers are never required to take control to safely operate the vehicle. Also known as autonomous or “driverless” cars, they combine sensors and software to control, navigate, and drive the vehicle.” For Thrun [3], an AV is “an unmanned vehicle that is capable of sensing its environment and navigating
without human input”. However, the standard way to discuss autonomous vehicles is to talk about “self-driving levels”, as defined by the SAE (Society of Automotive Engineers) [4]. The SAE, which is
an automobile standardization agency, divided the autonomous driving capacity of a vehicle into six levels, from the most basic systems to 100% autonomous driving. These levels help measure how advanced the technology of a certain autonomous car is. This has opened up numerous fields of
research and development that, although end up being interconnected, correspond to very diverse
areas.

In parallel to this evolution, the processes and procedures (function requirements and regulations) established for testing AV functions have also been developed and established over the previous decades. There, however, are concerns regarding the possible consequences of such a
technology, especially with regard to peoples’ safety, mechanical failures that may cause crash and the costs of such an incident. To guarantee that an AV is safe and to reduce costs, different scenarios
must be modelled and tested. Systematic testing of autonomous vehicles can be performed in Sensors 2019, 19, 648 2 of 29
simulation or in the physical world.

Physical tests offer real testing scenarios, and engineers can use
actual vehicles instead of models. However, regulations restrict the use of these vehicles in cities;
thus, to perform tests with real AVs, one needs access to expensive hardware and field tests, which
consume a considerable amount of time.

According to recently published reports [5,6], it is impossible
to perform empirical field tests that verify the safety of autonomous cars in a reasonable timeframe.

In this context, simulation, modelling and testing has the potential to fill the gap and enable rigorous,
controlled, and timely evaluation AV systems.

The research community recognizes three types of simulations [7]: live, virtual, and constructive
(LVC). A live simulation is simply an operational test, with sensors used to identify which systems
have been damaged by simulated firings, using real forces and real equipment. It is the closest
exercise to real use.

A virtual simulation ("X-in-the-loop") might test a complete system prototype  with stimuli produced either by a computer or otherwise artificially generated. This sort of exercise
is typical of a developmental test.

A constructive simulation is a computer-only representation of a
system or systems. Therefore, a simulation can vary from operational tests to a fully computer generated representation (i.e., no system components involved) of how a system will react to multiple
inputs. It can be used for several purposes: (1) The design and evaluation of operational and
development tests; (2) simulation, taking into account the levels of system aggregation, modeling
level of the individual components of a system (for example, the system software), or modeling level
of the system as a whole—to model a complete prototype and model multiple system interactions
(i.e. a RADAR component with the rest of the system) ...

However, as is exposed in [8], the first thing that must be taken into account is the type of
characteristics or systems that can be tested through simulation (see Figure 1). In this work, we focus
on the perception of autonomous vehicles as offering greater autonomy and complexity, emphasizing on their subsystems [9]: Environmental perception, and localization.

see fig 1:  enviro perception and localization;

mission planning; behavior planning; motion planning;

path trajectory and trajec tracking...

The paper is organized as follows: Section 2 and 3 detail the technologies habitually engaged to  capture information from the environment and to estimate position of the autonomous vehicles.
Section 4 discusses the current fusion technics used to combined information from the perception systems. Section 5 carries out an exhaustive collection of tools and platforms capable of performing
simulation tasks in the AV ambit. Section 6 shows the current state of legislation with regard to the legal aspects of autonomous driving and testing possibilities of AV in different countries. Section 7
presents our conclusions

2. Environment Perception System
An autonomous vehicle acquires knowledge of its surrounding in two stages.

The first stage consists of scanning the road ahead to detect possible changes in driving conditions (traffic lights and
signs, pedestrian crossing, and barriers, among others).

The second stage relates to the perception of
other vehicles.

This section presents the most representative sensors that make up the perception systems of AVs: Ultrasonic, RADAR, LiDAR, cameras, IMU, GNSS, and RTK. There are numerous informative
and scientific articles or books that show the types of sensors used in AVs, their applications, as well as their advantages and disadvantages. These works portray the sensors as black boxes, formed by a
set of inputs and outputs, without delving into the physical foundations of their operation. As a novel
aspect of this work, these sensors are presented from a point of view of the electromagnetic spectrum that they actively or passively use for their operation. This will allow researchers to acquire a deeper knowledge of the benefits and disadvantages of these sensors in degraded environments or adverse weather conditions. In Figure 2, the electromagnetic spectrum is divided into two scales:Wavelengths and frequencies. In addition, the spectral ranges used by the sensors analysed in this
work are shown.

fig 2:   sensors:  ultrasonic. radar;  thermal cameras;  lidar and Tof Flight;  visual cameras;

2.1. Ultrasonic Sensor
As its name indicates, ultrasonic sensors use sonic waves, in the range of 20 kHz to  40 kHz, generated by a magnetoresistive membrane, to measure the distance to an object. Its principle
of operation is based on the measurement of the time of flight (ToF) of the sonic wave from when it is emitted until the echo is received:
 =
2
× ToF (1)
The c velocity of the wave is in meters per second and ToF is time of flight in seconds.
These sensors are usually used in industrial environments for the measurement of height in
storage of all types of raw materials. In vehicles, they are used in parking systems or as short distance
measurement sensors at low speeds (Figure 3a and b). These low-cost sensors produce good results
when measuring distances with any material, independent of its colour, in dusty environments or in
adverse weather conditions (humidity or rain). Disadvantages of these sensors include a tendency to produce false positives by bouncing, and a blind zone (blanking) in the measurements, located
between the sender element of the sensor and the minimum range...

2.2. RADAR (Radio Detection and Ranging)
Radar systems work in wavelengths of the order of millimetres; these are used in a wide variety   of military and civil applications, such as aerial or terrestrial threat detection systems, shooting systems, and airports or meteorological systems. The emergence of smart vehicles and the need to
increase road safety have triggered the use of this type of device in the automotive sector. Radar  systems for intelligent vehicles work at frequencies of 24/77/79 GHz, known as millimetre wave radar
(MMW). The radar measures the distance between the emitter and the object by calculating the time  of flight of the emitted signal and the received echo. The radars not only allow the detection of the
distance to several targets, but are also capable of accurately supplying the direction and speed of the  targets. The new radars for vehicles use an array of micro antennas capable of generating a set of
lobes that allow improvement of the range and a processing system for the detection of multiple  targets (Figure 4a and b).

Range millimetre-wave RADAR is applied in Blind Spot Detection (BSD), Lane Change Assistant   (LCA), Rear Cross Traffic Alert (RCTA), Forward Cross Traffic Alert (FCTA) or radar video fusion.

Radar waves have higher penetrability because they offer good features in all weather conditions,
and can accurately detect short-range targets in front, to the side, and to the rear side of vehicle. For
this reason, they are used in several ADAS systems. RADAR can significantly improve vehicle safety
performance and reduce the decision-making burden of people at the wheel. Furthermore, it can be
installed besides the bumpers of the vehicle. Some disadvantages of this sensor type are the lack of precision, its reduced Field of View (FOV), and the fact that it can produce false positives due to
bouncing of the emitted signal .

(cameras go here)

4. Fusion Algorithms
Sensory fusion or data fusion aims to improve the measurement of two or more sources of data  from sensors, beyond the individual measurement of each of them. Sensory fusion is especially indicated when large amounts of disparate sensor data are produced.

Sensorial fusion applied to the  measurement of redundant data reduces the uncertainty of the measurement, increases the accuracy
and improves the integrity of the system, improving fault tolerance.

Obtaining a classification of the algorithms or fusion techniques is an arduous and difficult task due to multidisplinarity and the large number of case studies reported in the literature. In a review
carried out by [30], it is possible to find an extensive classification of fusion methods according to
different criteria such as: (1) relations between the input data sources; (2) input / output data types
and their nature; (3) abstraction level of the employed data; (4) different data fusion levels, and (5)
architecture types. In this review, we will stablish

three categories:

(1) estimation methods based on
Gaussian filters (e.g., Kalman filter (KF) or particle filters (PF)),

(2) probabilistic inference methods (i.e., Bayes theorem), and

(3) artificial intelligence methods based on machine learning algorithms.

Three categorizations are applied to both perception systems and location systems in autonomous vehicles. Following are examples that combine the merging of data from the different categorizations.
and different sensors of the perception and location systems.

4.1. Fusion Methods in Perception Systems
There are numerous works related to the sensors mentioned in this review that detect vehicles,  pedestrians, lanes, signs, and so on. In [31], the data supplied by a LiDAR and a stereo camera are
fused to improve a vehicle’s detection.

The method has two stages: (1) Hypothesis generation, where
a vehicle candidate is obtained, combining Haar features from depth maps with the AdaBoost classifier; and

(2) hypothesis verification, where a shape estimation of the candidates is calculated  using the LiDAR information and a support vector machine. The proposed fusion approach achieves  a lower false alarm rate in urban environments. A method that fuses the MMW RADAR with camera
information is presented in [32]. The work presents a collaborative fusion approach to achieve an optimal balance between vehicle detection accuracy and computational efficiency. The MMW radar
first detects the potential vehicle and provides a region of interest.

The vision processing module employs symmetry detection and active contour detection to identify the vehicle inside the region of
interest provided by the MMW radar. The experimental results show a 92.36% detection rate and 0% false alarm rate using a real-world dataset. In [33], the advantages of the fusion of LiDAR and RADAR
sensors are used in order to provide permanent precise spatial and dynamical data. The work  presents a real-time algorithm, which enables an autonomous car to comfortably follow other cars at
various speeds, while maintaining a safe distance. In [34], an unscented Kalman filter and Joint Probabilistic Data Association is used to fuse the data from a 2D LiDAR and camera information to
detect vehicles. The presented approach achieves an increase in safety for vehicle detection in single lane carriage-ways, where casualties are higher than for other road classes.

The increase in precision and robustness obtained with the fusion methods when redundant  information is available is especially relevant when the systems can produce injuries or death to people.

Pedestrian detection is a common area to use the fusion algorithms, as demonstrated by numerous works found on this topic. In [35], the distance data and the reflexivity of a 3D LiDAR are  fused through a set of 50 features to detect pedestrians. The features are composed by three subsets:
(1) Shape features obtained in XYZ projections of pedestrian cloud points, (2) Hu invariants moments  from XYZ projections and, (3) statistical features from the reflexivity data. The data fused was used to train an SVM. The pedestrian detection method obtained a higher classification rate when it was  compared with similar algorithms. In [36], a set of machine-learning algorithms (MLA) are tested  with different fusion schemes with the proposal of determining the best performance in pedestrian
detection. The experimental results provide a false positive rate, AUC (Area Under Curve ROC is a  metric to evaluate the performance of two classifiers), and provide an accuracy of 96.67%. An RGB
camera and a LiDAR are fused for pedestrian detection. LiDAR is used to evaluate the value of depth  perception for pedestrian detection. In the work, the detectors were trained on both input modalities
(from KITTI database) and various fusion strategies. The best performance was obtained by a late rescoring strategy that was designed to be sensitive to geometric context. The advance in GPUs has
allowed algorithm execution times that would have been unthinkable a few years ago (i.e., CNN); there are numerous projects that use convolutional neural networks (CNN) to obtain high grades of
data abstraction. In [37], multispectral information is used from a thermal camera and an RGB camera  (TRGB channels) and a multi-layer fused CNN to detect pedestrians under adverse illumination
conditions. The algorithm development on the basis of a multiple-layer fusion technique can  significantly reduce the detection miss rate. In [38], the distance data with reflectance information from 3D LiDAR is fused by means of a CNN. Distance and intensity raw data from LiDAR are
transformed to high-resolution (dense) maps, which allow direct implementation on CNNs, both as  single or multi-channel inputs. The results of the CNN were tested with the KITTI dataset Vision
Benchmark Suite [39] .

skipped here...

5. Simulation
Modelling and simulation are well-established tools for analysis, design, acquisition and training in the automotive domain.

Despite the heterogeneity of subsystems and disciplines involved
in the development of an autonomous vehicle [9], there are many simulation methods that together allow covering of the entire development process. V-model and its variants have become the most
common process models adopted in the automotive industry, guiding the development of systems on a variety of refinement levels with a multiple-stage validation and testing process [47]. By
applying virtual simulation technologies at different abstraction levels, several “X”-in-the-Loop (XIL)  (where “X” refers to any type of test included in the development process) testing setups can be
performed: Model-In-the-Loop (MIL), Software-In-the-Loop (SIL), and Hardware-in-the-Loop (HIL). Moreover, ISO 26262 itself does have some limited guidance on the use of simulation in verification
activities. The safety process of ISO 26262 is based on a safety V-model. As such, it is not straightforward to match with an agile development process, which is the natural choice for AV
development. Section 6, “Product Development at the Software level”, provides some directions on
verification of the software architecture design and recommends “simulation of dynamic parts of the
design”. Section 6 further notes that “software integration testing can be executed in different environments”, and lists as examples MIL tests, SIL tests, processor-in-the-loop tests, and HIL tests.

Although the number of tools for the simulation of autonomous vehicles has increased in recent years, it is very difficult to select the best tool for a specific development. Features like open-source,
multi-platform, personalization and documentation are desirable in all simulators. Furthermore,  regarding the simulation of such robotic systems, one must take into account all the aspects of physical implementation to further simplify the transition from virtual- to real-world scenarios.

The Agent paradigm has been introduced as well, as a way to address some of the current issues in autonomous-based driver behavior regarding its distribution capabilities, computing efficiency
and scalability [48,49]. However, these are not found in all the simulation tools. Some of the solutions
provide a series of toolchains to address the end-to-end process of an autonomous vehicle; others
only contemplate some of the subsystems or functionalities (Advanced Driver Assistance Systems
(ADAS), Simultaneous Localization And Mapping (SLAM), Driving, traffic, etc.). Most simulation
tools are compatible with programming languages such as C/C++, Perl, Python, Java, LabVIEW, URBI
or MATLAB
Below we present the different approaches that can be taken into account to select a simulator
for autonomous vehicles (vehicle tests, robotics, game engine, specific development), focusing on the
perception subsystem.

5.1. Vehicle Test Simulation
As previously mentioned, achieving high fidelity autonomous driving requires testing of autonomous characteristics in every possible scenario. The design, implementation, and testing of
vehicles in a wide range of use cases and in realistic traffic conditions are costly, time-consuming, complicated and, often, not reproducible. The integration of tests with the physical platform in these
cases are unnecessarily complex and often carried out in the last stage of the development process.

This makes prototype design, implementation, intensive testing and simulation with a real vehicle in
the simulation circuit the most effective way to verify and validate the design idea. The simulation of
Software-In-the-Loop (SIL) in the laboratory environment offers a safe way to prototype and
implement the control and algorithms of vehicles. The incorporation of a vehicle and real sensors in
the simulation loop (called Hardware-In-the-Loop or HIL) validates the design and reduces the time
required for system verification.

Simulators for Autonomous Vehicles, Perception Review, Spain, 2019

Attachments