The Industry Needs a Foolproof Way to Test Mass-Produced Autonomous Vehicles

The Industry Needs a Foolproof Way to Test Mass-Produced Autonomous Vehicles

Today, autonomous vehicles are being road tested, but their future depends on whether automakers can develop production line tests on mass-produced vehicles to ensure they’re safe.

Autonomous vehicles are currently being tested in various cities to test development of the vehicle’s self-driving skills. An example is San Francisco, which contains more people, cars, and cyclists that the self-driving vehicles must be aware of at any given time.

Going to an extreme, Waymo (formerly Google) built a 91-acre self-driving vehicle test track modeled after a small town in northern California, the largest in the country. Inside, the company built an entire fake city, complete with curbs and sidewalks, stop signs, traffic lights, a railroad crossing, and a turnabout. Here, the company can run tests with their autonomous vehicles that would be difficult to do in a real city.

The city environment can test object detection, prediction, and response functions. Stacked predictions—such as predicting that the car in front of the vehicle will brake because it’s about to get cut off by a cyclist, or that a car making a left turn in front of the car will yield to a pedestrian in a crosswalk—aren’t unusual. Similarly, stacked maneuvers and managing multiple road challenges together or in quick succession are often necessary.

Road tests depend on random events that may be encountered by the vehicle. The ideal testing situation, however, would be well-defined controlled events. The details of autonomous hardware and software will determine what should be tested in controlled events.

It’s All About the Sensors

A self-driving vehicle is essentially a computer on wheels. It’s a complex, sophisticated system that “sees” by using sensors to monitor its environment. The sensors feed information to a computer that combines the sensor data with high-definition map data to localize the vehicle. It detects and classifies objects, determines their location, and provides their speed and direction. It builds a three-dimensional model of the world that keeps track of important objects. This also predicts the objects’ future motion—pedestrians and trucks have different predicted movements.

As much as we as humans rely on these senses to navigate through the daily city jungle, autonomous vehicles rely just as heavily on the accuracy of their sensors. A vehicle must be able to react at speeds of even over 150 kmh. Essentially, then, vehicle sensors must work more accurately than human senses.

The figure shows the configuration of General Motors’ Cruise autonomous prototype. Prudent decision-making and operation are based on both what the sensors “see,” as well as what may be hidden from view. To perform these functions, a vehicle may use multiple LiDARs, cameras, and radars. Their combined data provides sensor diversity, allowing it to see complex environments. This helps, for example, identify pedestrians, vehicle types, and road details such as lane lines, construction zones, and signage. A complementary set of long-range sensors track high-speed objects, such as oncoming vehicles, and the short-range sensors provide detail about moving objects near the vehicle, such as pedestrians and bicycles.

Sensors used in General Motors’ Cruise autonomous vehicle.

Computer controls convert sensor commands for the actuators that control the steering, throttle, brake, and drive unit. The Controls function gives the self-driving system full vehicle maneuverability complete with stability, traction, and anti-lock brake systems that are fully active. And, cameras all scan both long- and short-range with views 360 degrees around the vehicle.

LiDAR provides highly precise feedback using laser measurements for both fixed and moving objects. It’s a surveying method that measures distance to a target by illuminating the target with pulsed laser light and measuring the reflected pulses with a sensor. Differences in laser return times and wavelengths can then be used to make digital 3D representations of the target.

Radar is complementary to LiDAR because it uses electromagnetic pulse measurements and can see solid objects that have low light reflectivity. LiDAR and radar inputs measure the speed of moving objects, allowing quick, confident determinations of speed. Cameras are also complementary to LiDAR because they measure the light intensity reflected off or emitted.

Mass-Production Testing

In mass production, you don’t have the luxury of road testing every car. Road testing cars is

really only the “learning curve” phase of autonomous-vehicle development. The data you get from road testing helps you design the cars. When you make millions of cars every year, you must develop a technique that tests the car in a few minutes. How do you test a car that can:

  • View the environment around it, in 360 degrees, day and night?
  • Identify pedestrians in a crosswalk?
  • Identify cars in front and the side of you?
  • Identify bicyclists on the road?
  • See an object darting suddenly into its path, and respond accordingly?
  • Maneuver through construction cones?
  • Yield to emergency vehicles?
  • React to avoid collisions?

In 2016, the U.S. Department of Transportation released a 15-point checklist to serve as a guideline for self-driving car manufacturers. Besides ethical and law issues, one of the 15 points refers to one of the most critical aspects car manufacturers face: “Automakers need to develop testing and validation methods that account for the wide range of technologies used in driverless cars.” Production testing of software and hardware reliability are among the most difficult issues confronting automakers. You can characterize vehicle production testing into key points, including:

  • Validation is the most critical aspect in quality assurance, covering various tests, such as regression, performance, functional, and security tests that ensure reliable product functionality.
  • Operational data must be analyzed and managed to avoid “data garbage” and reduce analysis times.
  • Cloud solutions for data communication may be necessary throughout the testing procedure.
  • Sensor accuracy and reliability are major criteria for every autonomous vehicle.
  • Testing systems should be based on open, flexible platforms that allow for testing of various sensor types and software, as well as the integration of various simulation scenarios.

Production testing relies on end-of-line tests that are short-duration, mechanical evaluations of vehicles that occur near the end of the production line. These tests can be automated inline operations that require minimal operator intervention, or offline quality audits when more detailed investigations are required. Specimens are subjected to a routine set of forces to determine pass or fail status, based on established manufacturing tolerances for squeak-and-rattle properties or other performance characteristics. The key with such testing depends on accuracy, repeatability, and ease-of-use to successfully discover problem issues while still keeping pace with the production line.

Wolfgang Klippel of the Dresden University of Technology described his view of production testing. “Testing a manufactured unit at the end of the assembly line is a critical step in the production process. Defective products or even those not matching specification limits closely enough must be separated from the functional units shipped to the customer. End-of-line testing assesses not only the quality of the product, but also the stability and yield of the production process. Reliable detection of non-functional units is the primary objective of the test, but reducing the rejection rate and maximizing the output is the ultimate goal.

“100% automatic testing replaces more and more subjective testing by human operators to shorten the production cycle and to improve the reproducibility and comparability of the results. However, objective measurements should provide a comprehensive assessment as sensitive as a human tester using his visual and aural senses. To fully compete with an experienced operator, the objective measurement instrument should also have learning capabilities to accumulate knowledge about physical causes of the fault. Furthermore, it should be capable of being integrated in automated lines, robust in a harsh and noisy environment, cost-effective, and simple to use.”

Regardless of the actual test procedure, all moving parts in the car must be checked, such as the steering wheel, steering system, windows, doors, windshield wipers, etc. Assuming it’s an electric vehicle, the traction motor and associated circuits should be checked, as well as all of the other motors in the car. In addition, all external and internal lighting should be tested along with the infotainment system. Everything associated with the advanced driver assistance systems (ADAS) should also be tested.

Testing Through Simulation?

One possible testing approach is to use a technique similar to a flight simulator—one that artificially recreates aircraft flight and the environment in which it flies, for pilot training, design, or other purposes. That includes replicating the equations governing how aircraft fly, how they react to applications of flight controls, the effects of other aircraft systems, and how the aircraft reacts to external factors such as air density, turbulence, wind shear, cloud, precipitation, etc.

Flight simulation is used for a variety of reasons, such as flight training (mainly of pilots), the design and development of the aircraft itself, and research into aircraft characteristics and control handling qualities. Although the flight simulator is intended to train pilots, maybe similar equipment could be used test autonomous vehicles.

For an autonomous vehicle, the simulator would have to artificially recreate the environment where it travels. It would include replicating the equations that govern how a car travels, how they react to applications of car controls, the effects of other cars, and how the car reacts to external factors such as pedestrians, other cars, weather conditions, etc.

Statistically significant assessments of skill transfer based on simulator testing and leading to handling an actual vehicle are difficult to make, particularly where motion cues are concerned. Large samples of driver opinions are required and many subjective opinions need to be aired, particularly by drivers not used to making objective assessments and responding to a structured test schedule.

Simulation is an established technique used in the man-machine systems for test evaluation. The principal task of a simulator is the creation of a dynamic representation of a vehicle’s behavior while allowing a test operator to control the simulation.

Simulation industry experience provides an insight into the technical disciplines that are combined to form a highly accurate representation of travel. Such disciplines include computer graphics, hardware and software engineering, man-machine systems, and mathematical systems modeling. Thus, the true art of test simulation is the successful integration of very specific areas to form an accurate representation of a vehicle.

Vehicle simulation must exercise both the hardware involved as well as the software. This is important because of the probable inclusion of artificial intelligence and machine learning in the autonomous vehicle. And, some vehicles employ generated maps that reflect the area in which the vehicle is traveling.

Questions arise as to how produce an environment similar to the real world? How do you simulate a child running across a street to retrieve a ball? How do you simulate an automobile accident in front of the vehicle? What about driving the right direction on a one-way street?

One possible approach would be to have a screen that presents some of these driving conditions. Presentations on the screen would have to appear moving and in three dimensions to allow the sensors in the tested vehicle to recognize the real driving situations. There may be a question as to whether the car’s sensors could function properly with a picture on a screen.

Another possibility would be to employ “dummies” of people and cars that can be moved to duplicate real conditions. This would be quite expensive in terms of cost and space required.

Also to be considered is that a simulator can cost several million dollars. Purchasing of a simulator is a long-term aspect of the simulator industry that reflects and impacts the automobile industry. For example, can you build one simulator that works with all vehicles? Or, can simulator design be adjusted to meet the requirements of different vehicle types?

Another possible testing procedure would be to run functional tests on all sensors employed in the car. Plus, all of the car’s software should be exercised to determine if it’s working properly. This may be difficult to do if the car uses artificial intelligence and machine learning. Even if this testing succeeds, you can’t assume everything will operate properly in the real-world environment.

Long-Term Support

Just because a car is shipped doesn’t mean the autonomous carmaker is through. For example, if there’s a software upgrade, how can the owner know that the car is safe? The same question applies if one of the sensors has to be replaced?

There must be some way where the owner can be able to tell if the car is safe after any changes. It’s different for a conventional car, which doesn’t have anywhere near the same amount of software and special sensors. Can the car dealer run a series of test to verify the autonomous car is safe to use? Or, the car might have some type of built-in self-test that’s supposed to verify the car is safe? The owner certainly doesn’t want to ship the car back to the factory, where it may have a more sophisticated test system.

You must realize that electronic product’s reliability is based on the number of components and their failure rate. When there’s a significant amount of software, reliability can suffer, but it’s difficult to tell how much it will be affected by changes. And, software upgrades don’t always do what they’re supposed to do.

No published evidence exists whereby autonomous-vehicle manufacturers have considered the long-term consequences of supporting that vehicle. There obviously is a learning curve for supporting an autonomous vehicle. It may take a while until the manufacturers understand the situation. Some questions for these automakers:

  • Do you realize the extent of coverage for an autonomous vehicle?
  • Do you understand the impact of software support?
  • How do you determine if a car is safe after a software upgrade?
  • How do you determine if a car is safe after one of its sensors is replaced?

Unfortunately, some car owners won’t be able to tell if their car is safe unless they have an accident.

Still to be determined is how insurance companies will react to autonomous cars. Will they have higher rates for autonomous vehicles? Will software upgrades and sensor replacement impact the insurance rate?

Another unknown is how will the state and federal government deal with widespread use of autonomous cars? Now they’re only looking at the road test of autonomous cars. Will they dictate testing that must be performed to determine if a car is safe?

And, finally, what about the legal issues for the owner and car manufacturer?

Hide comments

Comments

  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Publish