GIBSON ENVIRONMENT for Embodied Active Agents with Real-World Perception

You shouldn't play video games all day, so shouldn't your AI! We built a virtual environment that offers real world experience for learning perception.

Summary: Perception and being active (i.e. having a certain level of motion freedom) are closely tied. Learning active perception and sensorimotor control in the physical world is cumbersome as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given a fruitful rise to learning in simulation which consequently casts a question on transferring to real-world. We developed Gibson environment with the following primary characteristics:

I. being from the real-world and reflecting its semantic complexity through virtualizing real spaces,
II. having a baked-in mechanism for transferring to real-world (Goggles function), and
III. embodiment of the agent and making it subject to constraints of space and physics via integrating a physics engine (Bulletphysics).

Naming: Gibson environment is named after James J. Gibson, the author of "Ecological Approach to Visual Perception", 1979. “We must perceive in order to move, but we must also move in order to perceive” – JJ Gibson

Please see the website (http://env.gibson.vision/) for more technical details. This repository is intended for distribution of the environment and installation/running instructions.

Paper

"Embodied Real-World Active Perception", in CVPR 2018 [Spotlight].

Beta

This is the 0.3.1 release. Bug reports and suggestions for improvement are appreciated. change log file.

Dataset: To make the beta release lighter for the users, we are including a small subset (9) of the spaces in it. The full dataset includes 572 spaces and 1440 floors. It will be made available if we dont get a major bug report during the brief beta release.

Table of contents

• Installation * Quick Installation (docker) * Building from source * Uninstalling
• Quick Start * Web User Interface * Rendering Semantics * Advanced Guide * ROS Configuration
• Coding your RL agent
• Environment Configuration
• Goggles: transferring the agent to real-world

Installation

Installation Method

There are two ways to instal gibson, A. using our docker image (recommended) and B. building from srouce.

System requirements

The minimum system requirements are the following:

For docker installation (A): - Ubuntu 16.04 - Nvidia GPU with VRAM > 6.0GB - Nvidia driver >= 384 - CUDA >= 9.0, CuDNN >= v7

For building from the source(B): - Ubuntu >= 14.04 - Nvidia GPU with VRAM > 6.0GB - Nvidia driver >= 375 - CUDA >= 8.0, CuDNN >= v5

Download data

First, our environment assets data are available here. You can follow the installation guide below to download and set up them properly. gibson/assets folder stores necessary data (agent models, environments, etc) to run gibson environment. Users can add more environments files into gibson/assets/dataset to run gibson on more environments.

A. Quick installation (docker)

We use docker to distribute our software, you need to install docker and nvidia-docker2.0 first.

Run docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi to verify your installation.

You can either 1. build your own docker image or 2. pull from our docker image. 1 is recommended because you have the freedom to include more or less enviroments into your docker image. For 2, we include a fixed number of 8 environments (space1-space8).

1. Build your own docker image (recommended)

git clone https://github.com/StanfordVL/GibsonEnv.git
cd GibsonEnv/gibson
wget https://storage.googleapis.com/gibsonassets/assets_core_v2.tar.gz
tar -zxf assets_core_v2.tar.gz && rm assets_core_v2.tar.gz
cd assets
wget https://storage.googleapis.com/gibsonassets/dataset.tar.gz
tar -zxf dataset.tar.gz && rm dataset.tar.gz
### the commands above downloads assets data file and decpmpress it into gibson/assets folder
cd ../.. # back to GibsonEnv dir
docker build . -t gibson ### finish building inside docker

If the installation is successful, you should be able to run docker run --runtime=nvidia -ti --rm -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v <host path to dataset folder>:/root/mount/gibson/gibson/assets/dataset gibson to create a container. Note that we don't include dataset files in docker image to keep our image slim, so you will need to mount it to the container when you start a container.

1. Or pull from our docker image

docker pull xf1280/gibson:0.3.1

Notes on deployment on a headless server

We have another docker file that supports deployment on a headless server and remote access with TurboVNC+virtualGL. You can build your own docker image with the docker file Dockerfile_server. Instructions to run gibson on a headless server (requires X server running):

1. Install nvidia-docker2 dependencies following the starter guide.
2. Use openssl req -new -x509 -days 365 -nodes -out self.pem -keyout self.pem create self.pem file
3. docker build -f Dockerfile_server -t gibson_server . use the Dockerfile_server to build a new docker image that support virtualgl and turbovnc
4. docker run --runtime=nvidia -ti --rm -e DISPLAY -v /tmp/.X11-unix/X0:/tmp/.X11-unix/X0 -v <host path to dataset folder>:/root/mount/gibson/gibson/assets/dataset -p 5901:5901 gibson_server in docker terminal, start /opt/websockify/run 5901 --web=/opt/noVNC --wrap-mode=ignore -- vncserver :1 -securitytypes otp -otp -noxstartup in background, potentially with tmux
5. Run gibson with DISPLAY=:1 vglrun python <gibson example or training>
6. Visit your host:5901 and type in one time password to see the GUI.

If you don't have X server running, you can still run gibson, see this guide for more details.

B. Building from source

If you don't want to use our docker image, you can also install gibson locally. This will require some dependencies to be installed.

First, make sure you have Nvidia driver and CUDA installed. If you install from source, CUDA 9 is not necessary, as that is for nvidia-docker 2.0. Then, let's install some dependencies:

apt-get update 
apt-get install libglew-dev libglm-dev libassimp-dev xorg-dev libglu1-mesa-dev libboost-dev \
        mesa-common-dev freeglut3-dev libopenmpi-dev cmake golang libjpeg-turbo8-dev wmctrl \ 
        xdotool libzmq3-dev zlib1g-dev\

Install required deep learning libraries: Using python3.5 is recommended. You can create a python3.5 environment first.

conda create -n py35 python=3.5 anaconda 
source activate py35 # the rest of the steps needs to be performed in the conda environment
conda install -c conda-forge opencv
pip install http://download.pytorch.org/whl/cu90/torch-0.3.1-cp35-cp35m-linux_x86_64.whl 
pip install torchvision==0.2.0
pip install tensorflow==1.3

Clone the repository, download data and build

git clone https://github.com/StanfordVL/GibsonEnv.git
cd GibsonEnv/gibson
wget https://storage.googleapis.com/gibsonassets/assets_core_v2.tar.gz
tar -zxf assets_core_v2.tar.gz
cd assets
wget https://storage.googleapis.com/gibsonassets/dataset.tar.gz
tar -zxf dataset.tar.gz
#### the commands above downloads assets data file and decpmpress it into gibson/assets folder
cd ../.. #back to GibsonEnv dir
./build.sh build_local ### build C++ and CUDA files
pip install -e . ### Install python libraries

Install OpenAI baselines if you need to run training demo.

git clone https://github.com/fxia22/baselines.git
pip install -e baselines

Uninstalling

Uninstall gibson is easy. If you installed with docker, just run docker images -a | grep "gibson" | awk '{print $3}' | xargs docker rmi to clean up the image. If you installed from source, uninstall with pip uninstall gibson

Quick Start

First run xhost +local:root on your host machine to enable display. You may need to run export DISPLAY=:0.0 first. After getting into the docker container with docker run --runtime=nvidia -ti --rm -e DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v <host path to dataset folder>:/root/mount/gibson/gibson/assets/dataset gibson, you will get an interactive shell. Now you can run a few demos.

If you installed from source, you can run those directly using the following commands without using docker.

python examples/demo/play_husky_nonviz.py ### Use ASWD keys on your keyboard to control a car to navigate around Gates building

You will be able to use ASWD keys on your keyboard to control a car to navigate around Gates building. A camera output will not be shown in this particular demo.

python examples/demo/play_husky_camera.py ### Use ASWD keys on your keyboard to control a car to navigate around Gates building, while RGB and depth camera outputs are also shown.

You will able to use ASWD keys on your keyboard to control a car to navigate around Gates building. You will also be able to see the RGB and depth camera outputs.

python examples/train/train_husky_navigate_ppo2.py ### Use PPO2 to train a car to navigate down the hall way in Gates building, using visual input from the camera.

By running this command you will start training a husky robot to navigate in Gates building and go down the corridor with RGBD input. You will see some RL related statistics in the terminal after each episode.

python examples/train/train_ant_navigate_ppo1.py ### Use PPO1 to train an ant to navigate down the hall way in Gates building, using visual input from the camera.

By running this command you will start training an ant to navigate in Gates building and go down the corridor with RGBD input. You will see some RL related statistics in the terminal after each episode.

Web User Interface

When running Gibson, you can start a web user interface with python gibson/utils/web_ui.py python gibson/utils/web_ui.py 5552. This is helpful when you cannot physically access the machine running gibson or you are running on a headless cloud environment.

Rendering Semantics

Gibson can provide pixel-wise frame-by-frame semantic masks when the model is semantically annotated. As of now we have incorporated models from Stanford 2D3DS and Matterport 3D for this purpose, and we refer you to the original dataset's reference for the list of their semantic classes and annotations.

For detailed instructions of rendering semantics in Gibson, see semantic instructions. In the light beta release, the space 17DRP5sb8fy includes Matterport 3D style semantic annotation and space7 includes Stanford 2D3DS style annotation.

Agreement: If you choose to use the models from Stanford 2D3DS or Matterport 3D for rendering semantics, we ask you to agree to and sign their respective agreements. See here for Matterport3D and here for Stanford 2D3DS.

More Advanced Starting Guide

Starter Agents

Gibson provides a base set of agents:

To enable (optionally) abstracting away low-level control and robot dynamics for high-level tasks, we also provide a set of practical and ideal controllers for each agent.

Agent Name	DOF	Information	Controller
Mujoco Ant	8	OpenAI Link	Torque
Mujoco Humanoid	17	OpenAI Link	Torque
Husky Robot	4	ROS, Manufacturer	Torque, Velocity, Position
Minitaur Robot	8	Robot Page, Manufacturer	Sine Controller
JackRabbot	2	Stanford Project Link	Torque, Velocity, Position
TurtleBot	2	ROS, Manufacturer	Torque, Velocity, Position
Quadrotor	6	Paper	Position

Starter Code

More demonstration examples can be