Running latest Apollo with SVL Simulator

These instructions are tested after the last commit enabling SVL Simulator with latest Apollo master. Commits after that are assumed to work as well, but not guaranteed.

Big changes have recently been introduced in Apollo master; camera perception may not be working yet, and LiDAR perception is recently able to build but still unstable (and uses a lot of GPU memory making it challenging for Apollo to share an 8GB GPU with the SVL Simulator). Because of this we will be using the modular testing features of the SVL Simulator which allows the simulator to directly publish perception and traffic light messages.

For those who used our fork of Apollo 5.0 before: please note the new step to select the correct setup mode in Dreamview.

Table of Contents

Getting Started top#

This guide outlines the steps required to setup Apollo for use with the SVL Simulator. If you have not already set up the simulator, please do so first by following the instructions in Installing the SVL Simulator.

Prerequisites top#

  • Ubuntu 18.04 or later
  • NVIDIA graphics card (required for Perception)
    • NVIDIA proprietary driver (>=410.48) must be installed

Setup top#

Docker top#

Apollo is designed to run out of Docker containers. The image will mount this repository as a volume so the image will not need to be rebuilt each time a modification is made.

Installing Docker CE top#

To install Docker CE please refer to the official documentation.

NOTE Apollo does not work if the docker is started with sudo. We suggest following through with the post installation steps.

Installing the NVIDIA Container Toolkit top#

Before installing the NVIDIA Container Toolkit, make sure that you have the appropriate NVIDIA drivers installed. To test if the NVIDIA drivers are properly installed enter nvidia-smi in a terminal. If the drivers are installed properly an output similar to the following should appear.

    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 460.39       Driver Version: 460.39       CUDA Version: 11.2     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  GeForce GTX 1080    Off  | 00000000:02:00.0  On |                  N/A |
    | 35%   54C    P0    44W / 180W |   3048MiB /  8116MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    +-----------------------------------------------------------------------------+

The installation steps for the NVIDIA Container Toolkit are available here.

Cloning the Repository top#

Clone latest Apollo using the following command:

git clone https://github.com/ApolloAuto/apollo

Building Apollo and bridge top#

Now everything should be in place to build Apollo. Apollo must be built from the container. To launch the container navigate to the directory where the repository was cloned and enter:

./docker/scripts/dev_start.sh

This should launch the container and mount a few volumes. It could take a few minutes to pull the latest volumes on the first run.

To get into the container:

./docker/scripts/dev_into.sh

Build Apollo (optimized, not debug, with GPU support):

./apollo.sh build_opt_gpu

NOTE The Apollo build may fail on machines with less than 1GB of RAM per CPU core due to aggressive parallelization in the build, as discussed in Apollo issue 7719.

If the build fails, either re-start it until it succeeds, but if it continues to fail (especially when running the linker) then you'll need to address the low memory situation by either adding more memory to your build machine or enabling or increasing available swap space. If your Apollo build is crashing on a 16GB machine with little or no swap, try setting it to 16GB.

Adding a Vehicle to Apollo top#

  • In order to run Apollo with the simulator, the calibration files for the vehicle you use in the simulator must be available in Apollo for perception module to work correctly.
  • Currently, only calibration files for Lincoln 2017 MKZ are included in the latest Apollo. More calibration files may be added later.
  • To add a new vehicle to Apollo:
    • Create a new folder with the new vehicle's name under APOLLO_ROOT/modules/calibration/data.
    • Copy all calibration files of Lincoln 2017 MKZ to the folder.
    • Update vehicle parameters in vehicle_param.pb.txt if your new vehicle is different from Lincoln 2017 MKZ.
    • Update LiDAR calibration files in velodyne_params folder. If you have different configuration for LiDAR with respect to novatel, you need to update LiDAR's height and extrinsics.
    • Update camera calibration files in camera_params folder. You need to update camera intrinsics and extrinsics for your own camera and sensor configurations.
    • You may also need to update other sensor calibration files if you are using them on your vehicle.
  • After you add the new vehicle, you need to restart dreamview to refresh the vehicle list.
    • Inside Apollo docker:
      • bootstrap_lgsvl.sh stop && bootstrap_lgsvl.sh

Adding an HD Map to Apollo top#

  • An HD map in Apollo format corresponding to the environment you use in the simulator must be available in Apollo for modules like routing, planning, etc. to work.

  • Currently, only HD map files for Borregas Ave are included in the the latest Apollo at this time. More HD maps may be added later.

  • You can also download HD Maps for any map available in the Web UI Store (for example), and manually add them to /apollo/modules/map/data/.

  • Or you can export your own map file following Export Map Annotations, then generate routing_map and sim_map following these instructions.

  • In our Apollo 5.0 fork, we have a script generate_map.sh to help users generate map files. You can copy that script to APOLLO_ROOT/scripts/ folder to use it. For details, please follow here.

  • After you add the new map, you need to restart Dreamview to refresh the map list.

    • Inside Apollo docker:
      • bootstrap_lgsvl.sh stop && bootstrap_lgsvl.sh

Alternatively, you can analyze top while building, and decide how many jobs to enable to avoid running out of memory; then set a specific value in JOB_ARG, e.g. --jobs=6.

Setting Clock Mode in Apollo (Optional) top#

SVL Simulator has a clock sensor which outputs a simulated time that can be used as the reference time for Apollo. When used, message lag which may result from transporting large amounts of data between the Simulator and Apollo will not affect Apollo performance. In addition, the clock sensor allows Apollo to run with "simulator time" when used in "stepped simulation" mode as shown in Python API Quickstart Script 33.

To be able to use the clock sensor (which is part of the Apollo 6.0 (modular testing) sensor configuration), clock_mode must be set to MODE_MOCK in cyber.pb.conf.

Launching Apollo alongside the Simulator top#

Here we only describe only a simple case of driving from point A to point B using Apollo and the simulator.

To launch Apollo, first launch and enter a container as described in the previous steps.

  • To start Apollo:

    Note: you may receive errors about Dreamview not being build if you do not run the script from the /apollo directory.

./scripts/bootstrap_lgsvl.sh
  • Launch bridge (inside docker container):
./scripts/bridge.sh
  • Run the SVL Simulator outside of docker. See instructions for Running the Simulator
    • Create a Simulation using the Random Traffic runtime template and selecting the BorregasAve map and Lincoln2017MKZ vehicle with the Apollo 6.0 (modular testing) sensor configuration
    • Enter localhost:9090 as the Bridge Connection String (if apollo and the simulator are running on separate machines you will need to enter the IPv4 address of the machine running the bridge instead of localhost)
    • (Optional) Enable Traffic and Pedestrians
    • (Optional) Set the Time of Day and weather settings
    • Submit the Simulation
    • Select the created Simulation and click "Run Simulation"

  • Open Apollo Dreamview in a browser by navigating to: localhost:8888
    • NEW for Apollo Master: Select the Mkz Lgsvl setup mode, from the menu to the left of the vehicle menu.
    • Select the Lincoln2017MKZ LGSVL vehicle and Borregas Ave map in the top right corner.
    • Open the Module Controller tap (on the left bar).
    • Enable Localization, Transform, Planning, Prediction, Routing, and Control (there is no need to enable Perception or Traffic Light because the modular testing sensors directly publish perception and traffic light results).
    • Navigate to the Route Editing tab.
    • Select a destination by clicking on a lane-line and clicking Submit Route.
    • Watch the vehicle navigate to the destination.
  • To stop the docker container run the dev_start.sh stop script in apollo/docker/scripts in a new terminal (not in the docker container).
./dev_start.sh stop
  • If you are using ufw, it is easiest to completely disable the firewall to allow connections sudo ufw disable
    • If that is not possible, add the following rules:
      • These are required even if running the simulator and Apollo on the same machine
      • sudo ufw allow 8888
      • sudo ufw allow 9090

Adding a Vehicle top#

Only calibration files for Lincoln 2017 MKZ is included in the latest Apollo at this time. More calibration files map be added later.

Adding an HD Map top#

Only HD map files for Borregas Ave is included in the the latest Apollo at this time. More HD maps may be added later.

You can also download additional maps from here, and manually add them to /apollo/modules/map/data/.

Or you can export your own map from our map annotation tool, name it base_map.bin and then generate routing_map and sim_map following these Apollo map instructions.