This software is in development (TRL 6).

Knowledge Engine

Welcome to the source code repository of the Knowledge Engine. This README should help you understand what the Knowledge Engine is, and how to use it.

In short, the Knowledge Engine is an interoperability solution that wants to make it easier for different knowledge bases to communicate.

Knowledge base is a term to describe "the system that connects with the interoperability platform". It can be anything. Examples include, databases, sensors, or even GUIs.

Communication is made easier by using concepts from an ontology: a common domain-specific model that the knowledge bases will have to agree upon.

For example, if one knowledge base is interested in some kind of data (e.g., names of dogs: ?dog a ex:Dog . ?dog ex:hasName ?dogName .), and several other knowledge bases provide such data, the data is automatically gathered and merged when such a query is executed.

The Knowledge Engine consists of a number of components:

• A smart connector is used by knowledge bases to exchange data with other knowledge bases. The smart connector gives the knowledge base a way to (1) register the kind of knowledge they want to exchange and in which way, and (2) do the actual knowledge exchange.
• A Knowledge Engine runtime is a runtime in which one or more smart connectors can exist. These can exchange data with eachother, and, if configured correctly, also with smart connectors in other Knowledge Engine runtimes.
• (optional) A knowledge directory is used as a simple discovery mechanism for several Knowledge Engine runtimes to become aware of eachother's existence.

To quickly see how to use the Knowledge Engine through examples, see ./examples/.

Detailed documentation can be found here.

The rest of this README is structured as follows:

• Knowledge Engine
• Demonstration videos and tutorials
• Starting a Knowledge Engine runtime
  • Running with Docker
  • Running with Java
  • Advanced administration
  • Running with Java without the REST API
• Integrating with the Knowledge Engine
• Using the REST API
• Using the Java API
• Performance benchmark
• Current development
• Developer information
• Components
• Release steps
• Code conventions
• (advanced) Administering a Knowledge Engine runtime
  • Starting the Knowledge Engine in local mode
  • Starting the Knowledge Engine in distributed mode
  • Configuration

Demonstration videos and tutorials

This video gives a high-level introduction to the Knowledge Engine, with a simple demonstration of how we aimed to use it in the InterConnect project.

Following the previous video, this video shows the realisation, and explains more about a pilot in the Netherlands that uses the Knowledge Engine to control energy flexibility in a 22-story building with 160 appartments.

This video tutorial gives technical details about how to develop knowledge bases and connect them with the Knowledge Engine.

Starting a Knowledge Engine runtime

Starting a Knowledge Engine runtime can be done in several ways, with Docker, with Java, and in a more minimal way with Java.

^* Requires additional configuration

Running with Docker

The easiest way to start a Knowledge Engine runtime is with Docker:

docker run \
    -p 8280:8280 \
    ghcr.io/tno/knowledge-engine/smart-connector:1.5.0

The Knowledge Engine runtime is now available to use via the REST API at base URL http://localhost:8280/rest on your host machine. For usage instructions, see the section on using the REST API.

However, running it in the above way does not support data exchange with remote runtimes; it can only be used to exchange data with other smart connectors in the same runtime.

To interact with other runtimes, it needs additional configuration:

• An additional port mapping for the socket that listens for communication from other runtimes.
• ke.runtime.exposed.url: The URL via which the above socket is available for communication from the other runtime(s). You need to make sure the traffic is correctly routed to the container's port 8081 with a reverse proxy.
• kd.url: The URL on which to find the knowledge directory (to discover peers).

The configuration can be set as follows (note that the configuration properties below use underscores and capital letters)

docker run \
  -p 8280:8280 \
  -p 8081:8081 \
  -e KD_URL=https://knowledge-directory.example.org \
  -e KE_RUNTIME_EXPOSED_URL=https://your-domain.example.org:8081 \
  ghcr.io/tno/knowledge-engine/smart-connector:1.5.0

Running with Java

If you prefer not to use Docker, you can also use the JAR to run it directly in your JVM:

# Where the knowledge directory is located
export KD_URL=https://knowledge-directory.example.org
# Port where the runtime will listen for connections from other
# smart connectors.
export KE_RUNTIME_PORT=8081
# URL where the runtime will be available from other runtimes.
export KE_RUNTIME_EXPOSED_URL=https://your-domain.example.org:8081

# Start it. The argument (8280) denotes the port number at which it
# will listen for connections to the Knowledge Engine REST API.
java -jar -Dorg.slf4j.simpleLogger.logFile=smart-connector.log \
  smart-connector-rest-dist-1.5.0-with-dependencies.jar 8280

The JAR can be retrieved by compiling the project:

mvn clean package -DskipTests
# the relevant JAR will be in `./smart-connector-rest-dist/target/`

Advanced administration

For further advanced instructions on administering a Knowledge Engine runtime, refer to the advanced section of this document.

Running with Java without the REST API

In constrained environments, it may be preferable to use the Java API directly, and not use the REST API.

For this, you need to be in a Java project, import our packages, implement the KnowledgeBase interface, and start a smart connector with a SmartConnectorBuilder. An example of this can be found in the ./examples/ folder.

Integrating with the Knowledge Engine

Again, there are several options here: use the REST API, the Java API or the Knowledge Mapper^*.

^* Under development, and currently not available as open-source software.

Using the REST API

Assuming there is a REST API instance running at a known host, you can use these instructions to help you get started with making a client for it.

To make the client, it needs to talk HTTP, and conform to our API specification. With the API specification, you will be able to:

• Register your knowledge base via the /sc path.
• Register knowledge interactions for your knowledge base via the /sc/ki path.
• Trigger new proactive knowledge requests via the /sc/ask and /sc/post paths.
• Long-poll (GET) and respond to (POST) knowledge requests from the network via the /sc/handle path.

In the examples/rest-api folder, there is an example Docker Compose project with 3 knowledge bases that publish, store, and present sensor data through a single Knowledge Engine runtime. This example covers all four knowledge interaction types, but does not cover all features the REST API provides.

Using the Java API

In the Java API Example module, the Java API is used to share bindings through a POST knowledge interaction as they appear on an MQTT queue. Another knowledge base receives those bindings through a REACT knowledge interaction an prints them to the console.

Performance benchmark

A preliminary performance benchmark of the Knowledge Engine is available in this repository.

Based on this benchmark, the minimum requirements for the Knowledge Engine are: - i3-3040 Intel processor at 3.4 GHz - 8 Gbit RAM - 250Gb SSD drives

Of course, it highly depends on how you want to use the Knowledge Engine, because the reasoner, for example, increases these requirements.

Current development

The Knowledge Engine currently still undergoes further development. An integration with Eclipse Dataspace Components (EDC) is among the features in development, which is not stable in its current form and cannot be guaranteed to not crash frequently and contain bugs. It will be subject to many changes in the future. However, an example on the intended (future) use of this functionality is detailed in the example with a description of the setup and instructions on parameters to set and how to execute it.

Developer information

This section gives more detailed information about the project's structure, and is targeted towards developers who contribute code to the project.

Components

The Knowledge Engine project consists of the following Maven modules: - smart-connector - This is the implementation of the smart connector, with the Java developer API. For instructions on how to use it, refer to the documentation. The high-level design of the smart connector can be found in the wiki. - smart-connector-api - This module contains interfaces for the smart connector and other classes. It is made as a separate module so that it is easy to use different implementations of the interfaces. - smart-connector-rest-server - This module contains the REST API layer that is built on top of the Java Developer API. - smart-connector-rest-dist - A distribution of the server that provides the REST API layer for your smart connector(s), and uses the smart connector implementation from the smart-connector module. For instructions on how to use it, refer to the section below. For instructions on how to set it up, refer to this section. - admin-ui - A REST API which provides meta-data about smart connectors in a knowledge network. Can be used in an administration inferface for a knowledge network. It is implemented as a knowledge base that uses metadata of other knowledge bases. - reasoner - This module contains the reasoner specifically designed for the distributive nature of the knowledge engine. - knowledge-directory - This module contains the Knowledge Directory which is used to find other knowledge engine runtimes.

Release steps

These are instructions on what to do when we release a new version of the knowledge engine.

1. Update all relevant version references and make sure they are correct and non-SNAPSHOT:
   • this README.md file
   • all pom.xml files
   • openapi-sc.yaml version
   • Docker image tags in the Docker Compose examples.
2. Make a commit and push it to a new branch for the release.
3. Make a Pull Request and merge it into master after tests have succeeded.
4. Go to Actions on GitHub and trigger the workflow "Make draft release" with the new version as input.
5. When the workflow is finished, go to the newly generated draft release. Check the release notes, make any changes if necessary, and publish the release.
6. Prepare the next SNAPSHOT version and make a commit for that in a PR and merge into master.
7. openapi-sc.yaml
8. pom.xml
9. Leave the non-SNAPSHOT version in this README, and in the Docker Compose examples.
10. Inform mailing list(s) (and the blog) about the new release.

Code conventions

The code conventions of the knowledge-engine can be found in the /ide folder in the Eclipse IDE format. The format can often also