Pivotal Application Service > Butler

You are a platform operator working for a Fortune 500 enterprise. You've witnessed first-hand how the product development teams your team supports are super productive; happily invoking cf push, cf create-service and cf bind-service many times per day to deploy applications, create services and bind them to those applications.

This is great, except that over time, on your non-production foundations, as you've browsed organizations and spaces, you have noticed a large number of stopped application instances and orphaned services (i.e., those not bound to any applications).

Reaching out to each development team to tell them to clean-up has become a chore. Why not implement some automation that allows you a) to obtain snapshot and usage reports and b) define and enforce some house-keeping policies for your non-production foundations where applications and services are perhaps more volatile and c) easily handle multi-organization or system-wide use-cases like application instance scaling or stack changes?

This is where cf-butler has your back.

Table of Contents

• What does it do?
  • Tell me, don't show me
  • And what about Pivotal Telemetry Collector?
• Prerequisites
• Tools
• Clone
• How to configure
  • Managing secrets
  • Minimum required keys
    • Username and password
    • Single-sign on
  • General configuration notes
  • Using an external database
  • Managing policies
    • Hygiene Policies
    • Legacy policies
    • Endpoint policies
    • Query policies
  • To set the operations schedule
  • To discriminate user from service accounts
  • Filtering organizations
    • Blacklist
    • Whitelist
  • Integration w/ Operations Manager
• How to Build
  • Alternative build with MySQL support
• How to Run with Gradle
• How to Run with Docker
• How to deploy to Pivotal Application Service
  • with Username and password authorization
  • with SSO authorization
  • using scripts
• Endpoints
  • Operations Manager
  • Pivotal Network
  • Events
  • Metadata
  • Snapshot
  • Accounting
  • Policies
• Credits

What does it do?

Please take 5-10 mintues to view this short video demonstration to get a sense of what cf-butler can do.

Tell me, don't show me

Cf-butler exposes a number of self-service endpoints that perform house-keeping for your foundation. You define policies and an execution schedule. E.g., applications and service instances could be removed based on policy crtieria. Cf-butler also provides detail and summary snapshot reporting on all applications, service instances, user accounts, organizations and spaces. Lastly, cf-butler aspires to provide operators insight into the "freshness" of installed tiles, stemcells and buildpacks.

And what about Pivotal Telemetry Collector?

Pivotal Telemetry Collector supports collection of configuration data from Operations Manager, certificate data from Credhub, and usage data from Pivotal Application Service. Customers download and install a CLI from Pivotal Network. Typically, a Concourse pipeline is configured to automate collection. The result of collection is a foundation details tarball. Customers may opt to transmit this data to Pivotal.

Telemetry is also available for PCF Dev and Pivotal Container Service.

Note: the Pivotal Telemetry program is opt-in.

Cf-butler is configured and deployed as an application instance. Its capabilities overlap only on usage data collection from Pivotal Application Service. However, cf-butler performs other useful duties like a) snapshot usage reporting and b) policy registration and execution.

Prerequisites

Required

• Access to a foundation with Pivotal Application Service 2.6 or better installed
• and Pivotal Application Service admin account credentials

Optional

• Pivotal Network account credentials
• Pivotal Operations Manager admin account credentials

Tools

• git 2.20.1 or better
• JDK 11 or better
• cf CLI 6.41.0 or better
• uaac 4.1.0 or better

Clone

git clone https://github.com/pacphi/cf-butler.git

How to configure

Make a copy of then edit the contents of the application.yml file located in src/main/resources. A best practice is to append a suffix representating the target deployment environment (e.g., application-pws.yml, application-pcfone.yml). You will need to provide administrator credentials to Apps Manager for the foundation if you want the butler to keep your entire foundation tidy.

You really should not bundle configuration with the application. To take some of the sting away, you might consider externalizing and/or encrypting this configuration.

Managing secrets

Place secrets in config/secrets.json, e.g.,

{
  "PIVNET_API-TOKEN": "xxxxx"
  "CF_API-HOST": "xxxxx",
  "CF_USERNAME": "xxxxx",
  "CF_PASSWORD": "xxxxx",
}

We'll use this file later as input configuration for the creation of either a credhub or user-provided service instance.

Replace occurrences of xxxxx above with appropriate values

Minimum required keys

At a minimum you should supply values for the following keys

• cf.apiHost - a Pivotal Application Service API endpoint
• cf.tokenProvider - Pivotal Application Service authorization token provider, options are: userpass or sso
• pivnet.apiToken - a Pivotal Network legacy API Token, visit your profile

Based on choice of the authorization token provider

Username and password

• cf.username - a Pivotal Application Service account username (typically an administrator account)
• cf.password - a Pivotal Application Service account password

Single-sign on

• cf.refreshToken - the refresh token to be found within ~/.cf/config.json after your authenticate

General configuration notes

If you copied and appended a suffix to the original application.yml then you would set spring.profiles.active to be that suffix

E.g., if you had a configuration file named application-pws.yml

./gradlew bootRun -Dspring.profiles.active=pws

See the samples directory for some examples of configuration when deploying to Pivotal Web Services or PCF One.

For an exhaustive listing of all overridable configuration properties consult ButlerCfEnvProcessor.java.

Using an external database

By default cf-butler employs an in-memory H2 instance.

If you wish to configure an external database you must set set spring.r2dbc.* properties as described here.

Before you cf push, stash the credentials for your database in config/secrets.json like so

{
  "R2DBC_URL": "rdbc:postgresql://<server>:<port>/<database>",
  "R2DBC_USERNAME": "<username>",
  "R2DBC_PASSWORD": "<password>"
}

Replace place-holders encapsulated in <> above with real credentials

Or you may wish to cf bind-service to a database service instance. In this case you must abide by a naming convention. The name of your service instance must be cf-butler-backend.

DDL scripts for each supported database are managed underneath src/main/resources/db. Supported databases are: h2, mysql and postgresql.

A sample script and secrets for deploying cf-butler to Pivotal Web Services with an ElephantSQL backend exists for your perusal. If you're rather interested in MySQL as a backend, take a look at this version of secrets and the accompanying script.

Managing policies

Creation and deletion of policies are managed via API endpoints by default. When an audit trail is important to you, you may opt to set cf.policies.git.uri -- this property specifies the location of the repository that contains policy files in JSON format.

When you do this, you shift the lifecycle management of policies to Git. You will have to specify additional configuration, like

• cf.policies.git.commit the commit id to pull from
• if this property is missing the latest commit will be used
• cf.policies.git.filePaths an array of file paths of policy files

If you want to work with a private repository, then you will have to specify

• cf.policies.git.username
• cf.policies.git.password

one or both are used to authenticate. In the case where you may have configured a personal access token, set cf.policies.git.username equal to the value of the token.

Policy files must adhere to a naming convention where:

• a filename ending with -AP.json encapsulates an individual ApplicationPolicy
• a filename ending with -SIP.json encapsulates an individual ServiceInstancePolicy

A sample Github repository exists here.

Have a look at secrets.pws.json for an example of how to configure secrets for deployment of cf-butler to PAS integrating with the aforementioned sample Github repository.

On startup cf-butler will read files from the repo and cache in a database. Each policy's id will be set to the commit id.

Hygiene Policies

Hygiene policies are useful when you want to search for and report on dormant workloads, notifying both the operator and for each workload the author and/or his/her space compadres. Workloads are running applications and service instances that have not been updated in N or more days from the date/time of the policy execution.

Note: hygiene policy configuration has a special case where if the days-since-last-update property value is set to -1 then ALL workloads (minus the blacklist) are included in the respective notifications.

As mentioned previously the policy file must adhere to a naming convention

• a filename ending with -HP.json encapsulates an individual HygienePolicy

See additional property requirements in Query policies and the aforementioned sample Github repository.

Legacy Policies

Legacy policies are useful when you want to search for and report on applications deployed to a legacy stack (e.g., windows2012R2, cflinuxfs2) or service offering (e.g., using a product slug name like p-config-server, p-service-registry, p-mysql), notifying both the operator and for each application the author and/or his/her space compadres.

As mentioned previously the policy file must adhere to a naming convention

• a filename ending with -LP.json encapsulates an individual [LegacyPolicy](src/main/java/io/pivotal/