Attentive support

A simulation-based implementation of the attentive support robot introduced in the paper To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions. \ See the project website for an overview.

Setup

Python 3.8 - 3.11 \ Prerequisites for building the simulator workspace: g++, cmake, Libxml2, Qt5, qwt, OpenSceneGraph, Bullet Physics

Ubuntu 20

libxml2-dev, qt5-default, libqwt-qt5-dev, libopenscenegraph-dev, libbullet-dev, libasio-dev, libzmq3-dev, portaudio19-dev

Ubuntu 22

libxml2-dev, qtbase5-dev, qt5-qmake, libqwt-qt5-dev, libopenscenegraph-dev, libbullet-dev, libasio-dev, libzmq3-dev, portaudio19-dev

Fedora

cmake, gcc-c++, OpenSceneGraph-devel, libxml2, qwt-qt5-devel, bullet-devel, asio-devel, cppzmq-devel, python3-devel, portaudio

Clone this repo and change into it: git clone https://github.com/HRI-EU/AttentiveSupport.git && cd AttentiveSupport \ You can either run the setup script: bash build.sh or follow these steps: 1. Get the submodules: git submodule update --init --recursive 2. Create a build directory in the AttentiveSupport directory: mkdir -p build and change into it cd build 3. Install the smile workspace: cmake ../src/Smile/ -DCMAKE_INSTALL_PREFIX=../install; make -j; make install \ Note that if you have the Smile workspace installed somewhere else, you have to change the relative path in config.yaml accordingly. For details, check here 4. Install the Python dependencies: python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt 5. Make sure you have an OpenAI API key set up, see the official instructions 6. Enjoy 🕹️

Containerized Runtime

• Build the container: docker build -t localhost/attentive_support .
• Run the container with display support and set the OPENAI_API_KEY as environment variable.

podman:

podman run \
-e OPENAI_API_KEY=replace_me \
-e WAYLAND_DISPLAY \
--net=host \
-it \
localhost/attentive_support

docker (rootless):

docker run \
-e OPENAI_API_KEY=replace_me \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-it \
localhost/attentive_support

remote, rootless with internal ssh server:\ In certain scenarios it might not be possible to display the graphical window, e.g., when running docker rootless on a remote machine with X11. For these scenarios, the docker image can be built with the option docker build --build-arg WITH_SSH_SERVER=true -t localhost/attentive_support .. Include proxy settings as necessary. Start the image:

docker run \
  -it \
  -p 2022:22 \
  localhost/attentive_support

This starts an ssh server on port 2022 that can be accessed with username root and password hri: ssh -X root@localhost -p 2022. Run the example script:

export RCSVIEWER_SIMPLEGRAPHICS=True
export OPENAI_API_KEY=replace_me
/usr/bin/python -i /attentive_support/src/tool_agent.py

Usage

Running the agent

• Activate the virtual environment: source .venv/bin/activate
• Run the agent in interactive mode, from the AttentiveSupport directory: python -i src/tool_agent.py
• Provide commands: agent.plan_with_functions("Move the red glass to Felix")
• Reset
• The simulation: SIM.reset()
• The agent: agent.reset()

Customizing the agent

• Change the agent's character:
• Either via the system_prompt variable in gpt_config.py
• Or directly, note that this is not persistent: agent.character = "You are a whiny but helpful robot."
• Provide the agent with further tools:
• Define tools as Python functions in tools.py
• Make sure to use type hints and add docstrings in the Sphinx notation. This is important so that the function_analyzer.py can generate the function descriptions for openai automagically
• For inspiration, check out some more examples in src/tool_variants/extended_tools.py
• Change generic settings such as the model used and its temperature via gpt_config.py
• Note: The gpt_config.py file can either be changed directly, or the filename of a custom config file can be passed to the agent when running in interactive mode: python -i src/tool_agent.py --config=custom_config

Additional features

• Stop the robot mid-action: activate the simulation window, then press Shift + S
• Setting an agent as busy: set_busy("Daniel", "iphone5")
• Enable text to speech: enable_tts()
• Speech input; start talking after executing the command and press any key (or a specified push_key) to stop the speech input: agent.execute_voice_command_once()

Example

Running the simulation with "Move the red glass to Felix": \

For reproducing the situated interaction scenario run the following: - agent.plan_with_functions("Felix -> Daniel: Hey Daniel, what do we have to drink?") \ Robot should do nothing because Daniel is available to answer. - agent.plan_with_functions("Daniel -> Felix: We have two options, cola and fanta.") \ Robot should correct Daniel. - agent.plan_with_functions("Felix -> Daniel: Daniel, please hand me the red glass.") \ Robot should help because the red glass is out of reach for Daniel - Manually set Daniel to busy with the mobile: set_busy("Daniel", "iphone5") - agent.plan_with_functions("Felix -> Daniel: Daniel, could you fill some coca cola into my glass?") \ Robot should help as Daniel is busy. - agent.plan_with_functions("Daniel -> Felix: Felix, can you give me a full glass of the same, but without sugar?") \ Robot should help as Felix cannot see or reach the coke zero. - agent.plan_with_functions("Felix -> Robot: What do you know about mixing coke and fanta?") \ Robot should answer.