CF Acceptance Tests (CATs)

This suite exercises a Cloud Foundry deployment using the cf CLI and curl. It is scoped to testing user-facing, end-to-end features.

For example, one test pushes an app with cf push, hits an endpoint on the app with curl that causes it to crash, and asserts that we see a crash event in cf events.

Tests that won't be introduced here include things like basic CRUD of an object in the Cloud Controller. Such tests belong with the component they relate to.

These tests are not intended for use against production systems. They're meant for acceptance environments used by people developing Cloud Foundry's releases. While these tests attempt to clean up after themselves, there's no guarantee that they won't change the state of your system in an undesirable way. For lightweight system tests that are safe to run against a production environment, please use the CF Smoke Tests.

NOTE: Because we want to parallelize execution, tests should be written to be executable independently. Tests should not depend on state in other tests, and should not modify the CF state in such a way as to impact other tests.

1. Test Setup
   1. Install Required Dependencies
   2. Test Configuration
2. Test Execution
3. Explanation of Test Groups
4. Contributing

Test Setup

Prerequisites for running CATS

• Install golang >= 1.24.0. Set up your golang development environment, per golang.org.
• Install the cf CLI >= 8.5.0. Make sure that it is accessible in your $PATH.
• Install curl
• Check out a copy of cf-acceptance-tests. It uses Go modules, so there is no need to put it in $GOPATH.
• Install a running Cloud Foundry deployment to run these acceptance tests against. For example, bosh-lite.

Updating go dependencies

All go dependencies required by CATs are vendored in the vendor directory.

Make sure to have Golang 1.24+

In order to update a current dependency to a specific version, do the following:

cd cf-acceptance-tests
source .envrc
go get <import_path>@<revision_number>
go mod vendor

If you'd like to add a new dependency just run:

go mod tidy
go mod vendor

Test Configuration

You must set the environment variable $CONFIG which points to a JSON file that contains several pieces of data that will be used to configure the acceptance tests, e.g. telling the tests how to target your running Cloud Foundry deployment and what tests to run.

The following can be pasted into a terminal and will set up a sufficient $CONFIG to run the core test suites against a BOSH-Lite deployment of CF.

See example-cats-config.json for reference.

Only the following test groups are run by default:

include_apps
include_detect
include_routing
include_v3
include_app_syslog_tcp

The full set of config parameters is explained below:

Required parameters:

• api: Cloud Controller API endpoint, without scheme (HTTP/S) specified.
• admin_user: Name of a user in your CF instance with admin credentials. This admin user must have the doppler.firehose scope.
• admin_password: Password of the admin user above.
• apps_domain: A shared domain that tests can use to create subdomains that will route to applications also created in the tests, without scheme (HTTP/S) specified.
• skip_ssl_validation: Set to true if using an invalid (e.g. self-signed) cert for traffic routed to your CF instance; this is generally always true for BOSH-Lite deployments of CF.
• skip_dns_validation: Skip DNS validation for CF API and apps domain. Use true for proxy environments. Default:false

Optional parameters:

include_* parameters are used to specify whether to skip tests based on how a deployment is configured. * include_app_syslog_tcp: Flag to include the app syslog drain over TCP test group. * include_apps: Flag to include the apps test group. * readiness_health_checks_enabled: Defaults to true. Set to false if you are using an environment without readiness health checks. * include_cnb: Flag to include tests related to building apps using Cloud Native Buildpacks. Diego must be deployed and the CC API diego_cnb feature flag must be enabled for these tests to pass. The CF CLI version must be at least v8.14.0. * include_container_networking: Flag to include tests related to container networking. * credhub_mode: Valid values are assisted or non-assisted. See below. * credhub_location: Location of CredHub instance; default is https://credhub.service.cf.internal:8844 * credhub_client: UAA client credential for Service Broker write access to CredHub (required for CredHub tests); default is credhub_admin_client. * credhub_secret: UAA client secret for Service Broker write access to CredHub (required for CredHub tests). * include_deployments: Flag to include tests for the cloud controller rolling deployments. V3 must also be enabled. * include_detect: Flag to include tests in the detect group. * include_docker: Flag to include tests related to running Docker apps on Diego. Diego must be deployed and the CC API diego_docker feature flag must be enabled for these tests to pass. * include_file_based_service_bindings: Flag to include file-based service binding tests. For details, see RFC0030 * include_http2_routing: Flag to include the HTTP/2 Routing tests. * include_internet_dependent: Flag to include tests that require the deployment to have internet access. * include_isolation_segments: Flag to include isolation segment tests. * include_private_docker_registry: Flag to run tests that rely on a private docker image. See below. * include_route_services: Flag to include the route services tests. Diego must be deployed for these tests to pass. * include_routing: Flag to include the routing tests. * include_ipv6: Flag to include the IPv6 validation test group. * include_routing_isolation_segments: Flag to include routing isolation segments tests. See below. Cannot be run together with logging isolation segments tests. * include_security_groups: Flag to include tests for security groups. See below * dynamic_asgs_enabled: Defaults to true. Set to false if dynamic ASGs are disabled in the test environment. * comma_delim_asgs_enabled: Defaults to false. Set to true if comma delimited ASG destinations are enabled in the test environment. * include_services: Flag to include test for the services API. * include_service_instance_sharing: Flag to include tests for service instance sharing between spaces. include_services must be set for these tests to run. The service_instance_sharing feature flag must also be enabled for these tests to pass. * include_service_credential_binding_rotation: Execute tests for multiple service bindings. See RFC-0040 for details. This test requires CF CLI v8.18.0 or later. The backend must support at least 2 service bindings per app and service instance. * include_ssh: Flag to include tests for Diego container ssh feature. * include_sso: Flag to include the services tests that integrate with Single Sign On. * include_tasks: Flag to include the v3 task tests. include_v3 must also be set for tests to run. The CC API task_creation feature flag must be enabled for these tests to pass. * include_tcp_routing: Flag to include the TCP Routing tests. These tests are equivalent to the TCP Routing tests from the Routing Acceptance Tests. * tcp_domain: Domain that will be used for apps with TCP routes * include_user_provided_services: Flag to include test for user-provided services. * include_v3: Flag to include tests for the v3 API. * include_zipkin: Flag to include tests for Zipkin tracing. include_routing must also be set for tests to run. CF must be deployed with router.tracing.enable_zipkin set for tests to pass. * use_http: Set to true if you would like CF Acceptance Tests to use HTTP when making api and application requests. (default is HTTPS) * use_existing_organization: Set to true when you need to specify an existing organization to use rather than creating a new organization. * existing_organization: Name of the existing organization to use. * use_existing_user: The admin user configured above will normally be used to create a temporary user (with lesser permissions) to perform actions (such as push applications) during tests, and then delete said user after the tests have run; set this to true if you want to use an existing user, configured via the following properties. * keep_user_at_suite_end: If using an existing user (see above), set this to true unless you are okay having your existing user being deleted at the end. You can also set this to true when not using an existing user if you want to leave the temporary user around for debugging purposes after the test teardown. * existing_user: Name of the existing user to use. * existing_user_password: Password for the existing user to use. * artifacts_directory: If set, cf CLI trace output from test runs will be captured in files and placed in this directory. See below for more. * default_timeout: Default time (in seconds) to wait for polling assertions that wait for asynchronous results. * cf_push_timeout: Default time (in seconds) to wait for cf push commands to succeed. * long_curl_timeout: Default time (in seconds) to wait for assertions that curl slow endpoints of test applications. * broker_start_timeout (only relevant for services test group): Time (in seconds) to wait for service broker test app to start. * async_service_operation_timeout (only relevant for the services test group): Time (in seconds) to wait for an asynchronous service operation to complete. * test_password: Used to set the password for the test user. This may be needed if your CF installation has password policies. * timeout_scale: Used primarily to scale default timeouts for test setup and teardown actions (e.g. creating an org) as opposed to main test actions (e.g. pushing an app). * isolation_segment_name: Name of the isolation segment to use for the isolation segments test. * isolation_segment_domain: Domain that will route to the isolated router in the isolation segments and routing isolation segments tests. See below * include_tcp_isolation_segments: Flag to include the TCP Routing tests on Isolation Segments. These tests are equivalent to the TCP Routing tests from the Routing Acceptance Tests. * isolation_segment_tcp_domain: Domain that will be used for isolated apps with TCP routes * private_docker_registry_image: Name of the private docker image to use when testing private docker registries. See below * private_docker_registry_username: Username to access the private docker repository. See below * private_docker_registry_password: Password to access the private docker repository. See below * unallocated_ip_for_security_group: An unused IP address in the private network used by CF. Defaults to 10.0.244.255. See below

• staticfile_buildpack_name See below
• java_buildpack_name See below
• ruby_buildpack_name See below
• nginx_buildpack_name See below
• nodejs_buildpack_name See below
• go_buildpack_name See below
• r_buildpack_name See below
• binary_buildpack_name See below
• cnb_nodejs_buildpack_name See below

• python_buildpack_name: python_buildpack See below

• include_windows: Flag to include the tests that run against Windows cells.

• use_windows_test_task: Flag to include the tasks tests on Windows cells. Default is false.
• use_windows_context_path: Flag to include the Windows context path routing tests. Default is false.

• windows_stack: Windows stack to run tests against.

• include_service_discovery: Flag to include test for the service discovery. These tests use apps.internal domain, which is the default in cf-networking-release. The internal domain is currently not configurable.

• stacks: An array of stacks to test against. Currently cflinuxfs4 and cflinuxfs5 stacks are supported. Default is [cflinuxfs4].

• include_volume_services: Flag to include the tests for volume services. The following requirements must be met to run this suite: tcp-routing must be deployed.

• volume_service_name: The name of the volume service provided by the volume service broker.
• volume_service_plan_name: The name of the plan of the service provided by the volume service broker.
• volume_service_create_config: The JSON configuration that is used when volume service is created.
• volume_service_bind_config: The JSON configuration for the volume service binding configuration.

There is a Golang utility `bin/catsconfig