Python Test Cases

Video#


Introduction#

Test Case mode is a new feature in the SVL Simulator which executes test cases written in Python using the LGSVL Python API.

Python scripts can be used to implement many different kinds of tests to run in the SVL Simulator.

Among the general use-cases of Python Test Case mode are:

  • Generating and testing random variations of various driving test cases
  • Performing automated (e.g. acceptance) tests based on a set of repeatable test cases
  • Performing automated regressions tests

Python Test Case mode provides the ability to configure and execute several sample test cases available under "Available from Others" view in Simulations, using Apollo for Autonomous Driving.

Refer to Python Test Case Simulation for an illustrated step-by-step walk through on selecting and configuring a python test case simulation.

This document describes how to set up and run these test cases with Apollo.

For example, one provided example is a "cut-in" test case which is performed on the San Francisco map (shown below).

Another provided example is a "sudden braking" test case which is performed on the SingleLaneRoad map (shown below).

Additional examples demonstrate pedestrian-crossing test case which can be performed on the Straight1LanePedestrianCrosswalk map and the red-light runner test case which is performed on the Borregas Ave map.

(Screenshot of the "cut-in" test case on the San Francisco map, above)

(Screenshot of the "sudden-braking" test case on the SingleLandRoad map, above)

Requirements#

Python Test Case Mode has these dependencies:

  1. SVL Simulator -- the release tag of the simulator should match the release tag of Python Runner to ensure compatibility
  2. Apollo 5.0 lgsvl fork
  3. Docker and Docker-compose

Downloading and launching SVL Simulator top#

The Simulator release can be downloaded as a prebuilt binary from the github release page.

Download and extract the zip archive at the desired location. In the unzipped directory, run the executable file named simulator to launch the SVL Simulator.

The main window of the simulator will open. Click on the "Link to Cloud" or "Open Browser" button to launch the simulator Web UI in a web browser. The Web UI controls all aspects of the simulator.

You will need to create an account to login to the Web UI. Once logged in you will be able to link your simulator into a cluster, select maps, configure vehicles, and create simulations to run in the simulator.

For more information on initial setup of the simulator, refer to the Installing Simuator and Link to Cloud sections of the Running SVL Simulator documentation for an illustrated step-by-step walk through of this process.

Installing and Building Apollo 5.0 top#

Please follow the instructions on the SVL Simulator documentation website to for Running Apollo 5.0 with SVL Simulator if you have not already done so.

This will involve installing Docker CE, Nvidia Docker and the LGSVL Docker image, then cloning the lgsvl fork of the Apollo 5.0 sources, and finally building Apollo and the bridge.

Important: Confirm that Apollo is able to drive the car to a selected destination when running in a "Random Interactive" simulation before attempting to use Test Case Mode. Refer to Random Interactive Simulation for instructions on setting up a random interactive simulation you can use to test Apollo functionality with the simulator.

Python Test Case Mode workflow#

  1. Configure Test Case simulation in Web UI
  2. Launch Apollo, configure Dreamview, and enable modules
  3. Start SVL Simulator in Test Case mode
  4. Configure ego destination in Dreamview

Configure Test Case simulation in Web UI (if needed) top#

From the simulator Web UI, you will need to locate in the Store, a map and vehicle, and add each of them to your library.

Refer to the Store for information on how to add the "SanFrancisco" map, "Lincoln2017MKZ" vehicle and sensors needed for "Apollo 5.0 (full analysis)" sensor configuration on this vehicle.

Refer to Python Test Case Simulation for an illustrated step-by-step walk through on selecting and configuring a Python test case simulation.

Note: The public "Cut-in Scenario" test case simulation should be ready to use if Apollo is running on the same machine as the simulator but if Apollo is running elsewhere, be sure to update the BRIDGE_HOST as described in the Python Test Case Simulation walk through.

Launch Apollo, Configure Dreamview, and Enable Modules top#

  1. Open a terminal, change directory to the apollo-5.0 project (lgsvl fork: https://github.com/lgsvl/apollo-5.0), and type:

    • ./docker/dev_start.sh to start apollo container
    • ./docker/dev_into.sh to enter the container
    • Make sure you have built Apollo with GPU enabled options; for more details, refer to Running Apollo 5.0 with SVL Simulator.
    • bootstrap.sh && bridge.sh & to start the Apollo Dreamview Web UI and cyber_bridge.

  2. Open browser, enter address localhost:8888 to load Dreamview, and select Lincoln2017MKZ from the vehicle menu.

    • Be sure to also select the correct map in Dreamview for the desired test case:

      • For the Cut In scenario, select San Francisco from the map menu.
      • For the Pedestrian Crossing scenario, select Straight 1 Lane Pedestrian Crosswalk from the map menu.
      • For the Red Light Runner scenario, select Borregas Ave from the map menu.
      • For the Sudden Braking scenario, select Single Lane Road from the map menu.

    • Enable required modules in the Module Controller view before using the Python Runner including Camera, Localization, Perception, Planning, Prediction, Routing, Traffic Light, and Transform.

    • Enable the Control module in the Module Controller view as well.

Note: The ego vehicle will not drive until a destination is selected and a routing request is made in the Route Editing view. However, a destination cannot be selected until the simulation is started and Apollo is able to localize the ego on the map.

Note: When changing the map selection in Dreamview, all modules will be disabled and will need to be re-enabled.

Start SVL Simulator in Test Case mode top#

  1. In the Simulations tab in the Web UI locate the Cut-in Scenario scenario simulation.
  2. Then click the red "Run Simulation" button under the Cut-in Scenario to start the simulation.

Note: The ego vehicle will not drive until a destination is selected and a routing request is made in the Route Editing view. However, a destination cannot be selected until the simulation is started and Apollo is able to localize the ego on the map.

Configure Ego Destination in Dreamview top#

After starting the simulation, Apollo should receive sensor information and localize the car onto the selected map.

In Dreamview, switch to Route Editing view, zoom out, scroll if needed, and click to set the destination in the same ego lane before the next intersection. Then click "Send Routing Request" to plan the route.

In Module View make sure all required modules are still enabled.

The ego vehicle should begin driving. The NPC should cut in front of the ego vehicle, and the ego vehicle should slow or stop to avoid hitting the NPC.

This test case can be repeated by clicking the red "Run" (triangle/play) button in the Web UI. Since the destination and route plan are already configured, re-starting the simulation will reset the ego start location and Apollo should immediately being driving.

Example Test Cases#

There are several example test cases provided in this preview release. Each test case has a required map, and a suggested destination, as summarized in the following table:

Test Case Vehicle Sensor configuration Map Destination
cut-in.py Lincoln2017MKZ Apollo 5.0 (full analysis) San Francisco same lane before next intersection
ped-crossing.py Lincoln2017MKZ Apollo 5.0 (full analysis) Straight1LanePedestrianCrosswalk end of the map
red-light-runner.py Lincoln2017MKZ Apollo 5.0 (full analysis) Borregas Ave other side of intersection
sudden-braking.py Lincoln2017MKZ Apollo 5.0 (full analysis) SingleLaneRoad end of the map

Refer to the Store for information on how to add the "SanFrancisco" map, "Lincoln2017MKZ" vehicle and sensors needed for "Apollo 5.0 (full analysis)" sensor configuration.

Refer to Python Test Case Simulation for an illustrated step-by-step walk through on selecting and configuring a python test case simulation.

Note that each map (environment) must be available in the Simulator Maps view, and the corresponding Apollo HD map must be selected in Dreamview.

Also note that changing maps in Dreamview will disable the modules so you will need to re-enable them as described above.

Details for each of the test cases are provided below.

Cut-in Test Case top#

On a two lane road, an NPC cuts in front of ego vehicle from an adjacent lane while maintaining previous speed, with a small enough distance in front of ego vehicle such that ego vehicle will need to react by either changing speed (braking) or turning or swerving away.

This test case runs on the San Francisco map with the Lincoln MKZ ego vehicle driven by Apollo.

Be sure that the San Francisco map is selected in Dreamview, and that all required modules are enabled.

To run the Cut-in test case, select the "Cut In" test case file as described above.

Then switch to the Route Editing view of Dreamview. Scroll (or zoom out) and click to set a destination in the same ego lane before the next intersection. Then click Send Routing Request and return to the Module Controller view to confirm that Control (and other modules) are enabled.

Pedestrian Crossing Test Case top#

A pedestrian begins crossing a crosswalk as the ego vehicle approaches the intersection. The ego vehicle will need to slow down or come to a stop to wait for the pedestrian to finish crossing.

This test case runs on the Straight1LanePedestrianCrosswalk map with the Lincoln MKZ ego vehicle driven by Apollo.

Be sure that the Straight1LanePedestrianCrosswalk map is selected in Dreamview, and that all required modules are enabled.

To run the Cut-in test case, select the "Pedestrian Crossing" test case file as described above.

Then switch to the Route Editing view of Dreamview. Scroll (or zoom out) and click to set a destination at the end of the map. Then click Send Routing Request and return to the Module Controller view to confirm that Control (and other modules) are enabled.

Red Light Runner Test Case top#

While the ego vehicle attempts to cross a 4-way intersection with a green traffic light, an NPC crossing the intersection from the right runs a red light and crosses the intersection. The Ego must react by braking, stopping, or turning in order to avoid a collision.

This test case runs on the Borregas Ave map with the Lincoln MKZ ego vehicle driven by Apollo.

Be sure that the Borregas Ave map is selected in Dreamview, and that all required modules are enabled.

To run the Cut-in test case, select the "Red Light Runner" test case file as described above.

Then switch to the Route Editing view of Dreamview. Scroll (or zoom out) and click to set a destination on the other side of the intersection. Then click Send Routing Request and return to the Module Controller view to confirm that Control (and other modules) are enabled.

Sudden Braking Test Case top#

The ego vehicle follows an NPC vehicle that is traveling at the speed limit. The NPC suddenly brakes and comes to a stop. The ego vehicle will need to greatly reduce its speed to avoid a collision. After a few seconds, the NPC speeds back up gradually to the speed limit.

This test case runs on the SingleLaneRoad map with the Lincoln MKZ ego vehicle driven by Apollo.

Be sure that the Single Lane Road map is selected in Dreamview, and that all required modules are enabled.

To run the Cut-in test case, select the "Sudden Braking" test case file as described above.

Then switch to the Route Editing view of Dreamview. Scroll (or zoom out) and click to set a destination at the end of the map. Then click Send Routing Request and return to the Module Controller view to confirm that Control (and other modules) are enabled.

Known issues top#

  • The NPC usually waits until the ego vehicle starts driving but in some cases it might start before a route has been requested for the ego. In that case wait for the simulation to finish and then run it again while leaving the modules enabled and the route to the destination already specified.
  • If errors appear due to previously-downloaded (maps or vehicle) asset bundles from a previous version the Simulator, delete the Simulator persistent data folder located at /home/[username]/.config/unity3d/LGElectronics/SVLSimulator.
  • If Straight 1 Lane Pedestrian Crosswalk is not available in the Dreamview map menu, try updating your apollo-5.0 sources (from the lgsvl fork), then stop and re-start Apollo, and re-load Dreamview.
  • The AD stack (e.g. Apollo) is not currently controlled by the Python Runner, so you will need to manually set a destination in Dreamview.
  • You can control the AD stack through your Python Script using the python dreamview-API module. Dreamview-API module only supports Apollo at the moment. Using these APIs, you can configure hd_map and vehicle settings in Apollo Dreamview, enable/disable Apollo modules, set destination, etc..