github.com/GoogleCloudPlatform/datacatalog-tag-engine @v3.1.1

Tag Engine 3.0

This is the Dataplex branch for Tag Engine. Tag Engine v3 is the newest flavor of Tag Engine that is compatible with both Data Catalog and Dataplex. It is based on the Cloud Run branch and therefore supports VPC-SC, user authentication, role based access control.

This branch supports creating and updating aspects in bulk through the Tag Engine API. You can create aspects from CSV files on BigQuery, Cloud Storage, Spanner, and Cloud SQL resources using the import config type. You can also create dynamic aspects sourced from BigQuery metadata on BigQuery resources (columns, tables, views, datasets) through the dynamic table and column config types. See Part 2 for more details on how this all works.

If you're new to Tag Engine, it is an open-source extension to Dataplex. It runs on Google Cloud on top of Cloud Run and Firestore. It allows you to create structured metadata in bulk, sourced from either CSV or BigQuery. All the metadata created by Tag Engine is stored in the catalog entries of Dataplex and Data Catalog. The metadata is attached to the catalog entries of datasets, tables, views, files, and fields that exist in BigQuery, Cloud Storage, Cloud SQL or Spanner. For more information on Tag Engine's core functions, see sequence diagrams.

This README file describes the deployment steps, testing procedures, and code samples. It is organized into 6 sections:

• Part 1: Deploying Tag Engine v3

• Part 2: Testing your Tag Engine API Setup with Dataplex API

• Part 3: Testing your Tag Engine API Setup with Data Catalog API

• Part 4: Testing your Tag Engine UI Setup with Data Catalog API

• Part 5: Troubleshooting

• Part 6: Code Samples

Part 1: Deploying Tag Engine v3

Tag Engine v3 (just like Tag Engine v2) comes with two Cloud Run services. One service is for the Tag Engine API (tag-engine-api) and the other is for the UI (tag-engine-ui).

Both services use access tokens for authorization. The API service expects the client to pass in an access token when calling the API functions (gcloud auth print-identity-token) whereas the UI service uses OAuth to authorize the client from the front-end. Note that a client secret file is required for the OAuth flow.

Follow the steps below to deploy Tag Engine with Terraform.

Alternatively, you may choose to deploy Tag Engine with gcloud commands instead of running the Terraform.

1. Create (or designate) two service accounts:

2. A service account that runs the Tag Engine Cloud Run services (both API and UI). This account is referred to as TAG_ENGINE_SA.

3. A service account that sources the metadata from BigQuery or Cloud Storage, and then performs the tagging in Data Catalog. This account is referred to as TAG_CREATOR_SA.

See Creating Service Accounts for more details.

Why do we need two different service accounts? The key benefit of decoupling them is to allow individual teams to have their own Tag Creator SA. This account has permissions to read specific data assets in BigQuery and Cloud Storage. For example, the Finance team can have a different Tag Creator SA from the Finance team if they own different data assets. The Tag Engine admin then links each invoker account (either service or user) to a specific Tag Creator SA. Invoker accounts call Tag Engine through either the API or UI. This allows the Tag Engine admin to run and maintain a single instance of Tag Engine, as opposed to one instance per team.

1. Create an OAuth client:

Open API Credentials.

Click on Create Credentials and select OAuth client ID and choose the following settings:

Application type: web application

Name: tag-engine-oauth

Authorized redirects URIs: Leave this field blank for now.
Click Create

Download the credentials as te_client_secret.json and place the file in the root of the datacatalog-tag-engine directory

Note: The client secret file is required for establishing the authorization flow from the UI.

1. Create a new GCS bucket for CSV imports. Remember GCS bucket names are globally unique. For example: gsutil mb gs://$(gcloud config get-value project)-csv-import

2. Set the Terraform variables:

Open deploy/variables.tf and change the default value of each variable.

Save the file.

Alternatively, create a new file, named deploy/terrform.tfvars and specify your variables values there.

1. Run the Terraform scripts:

   NOTE: The terraform script will run with the default credentials currently configured on your system. Make sure that your current user has the required permissions to make changes to your project(s), or set new credentials using the GOOGLE APPLICATION_CREDENTIALS environment variable.

   cd deploy terraform init terraform plan terraform apply

   When the Terraform finishes running, it should output two URIs. One for the API service (which looks like this https://tag-engine-api-xxxxxxxxxxxxx.a.run.app) and another for the UI service (which looks like this https://tag-engine-ui-xxxxxxxxxxxxx.a.run.app).

2. The terraform script has created the tag engine configuration file (datacatalog-tag-engine/tagengine.ini). Open the file and verify the content, modifying if needed: TAG_ENGINE_SA TAG_CREATOR_SA TAG_ENGINE_PROJECT TAG_ENGINE_REGION FIRESTORE_PROJECT FIRESTORE_REGION FIRESTORE_DATABASE BIGQUERY_REGION FILESET_REGION SPANNER_REGION CLOUDSQL_REGION ENABLE_AUTH OAUTH_CLIENT_CREDENTIALS ENABLE_TAG_HISTORY TAG_HISTORY_PROJECT TAG_HISTORY_DATASET ENABLE_JOB_METADATA JOB_METADATA_PROJECT JOB_METADATA_DATASET CLONE_TAGS RETIRE_TAGS

A couple of notes:

• The variable ENABLE_AUTH is a boolean. When set to True, Tag Engine verifies that the end user is authorized to use TAG_CREATOR_SA prior to processing their tag requests. This is the recommended value.

• The tagengine.ini file also has two additional variables, INJECTOR_QUEUE and WORK_QUEUE. These determine the names of the cloud task queues. You do not need to change them. If you change their name, you need to also change them in the deploy/variables.tf.

• The CLONE_TAGS and RETIRE_TAGS are new variables. When set to True, CLONE_TAGS copies the tags to aspects at job execution time without the need for a separate Tag Engine configuration to create the aspects. When CLONE_TAGS and RETIRE_TAGS are both set to True, the tags don't get created, only the aspects do. For more details on these functions, please consult the documentation.

Part 2: Testing your Tag Engine API setup with Dataplex API

1. Create the sample data-governance aspect type:

   git clone https://github.com/GoogleCloudPlatform/datacatalog-templates.git cd datacatalog-templates python create_aspect_type.py $DATAPLEX_PROJECT $DATAPLEX_REGION aspect_types/data-governance.yaml

2. Grant permissions to invoker account (user or service):

   Depending on how you are involving the Tag Engine API, you'll need to grant permissions to either your service account or user account (or both).

   If you'll be invoking the Tag Engine API with a user account, authorize your user account as follows:

   ``` gcloud auth login

   export INVOKER_USER_ACCOUNT="username@example.com"

   gcloud iam service-accounts add-iam-policy-binding $TAG_CREATOR_SA \ --member=user:$INVOKER_USER_ACCOUNT --role=roles/iam.serviceAccountUser --project=$DATA_CATALOG_PROJECT

   gcloud run services add-iam-policy-binding tag-engine-api \ --member=user:$INVOKER_USER_ACCOUNT --role=roles/run.invoker --project=$TAG_ENGINE_PROJECT --region=$TAG_ENGINE_REGION ```

   If you are invoking the Tag Engine API with a service account, authorize your service account as follows:

   ``` export INVOKER_SERVICE_ACCOUNT="tag-engine-invoker@.iam.gserviceaccount.com"

   gcloud iam service-accounts add-iam-policy-binding $TAG_CREATOR_SA \ --member=serviceAccount:$INVOKER_SERVICE_ACCOUNT --role=roles/iam.serviceAccountUser

   gcloud run services add-iam-policy-binding tag-engine-api \ --member=serviceAccount:$INVOKER_SERVICE_ACCOUNT --role=roles/run.invoker --region=$TAG_ENGINE_REGION
   ```

   Very important: Tag Engine requires that these roles be directly attached to your invoker account(s).

3. Run the following commands to create Dataplex aspects from CSV files:

   Copy examples/configs/import/sample_data/bigquery_table.csv to your own GCS bucket.

   Open examples/configs/import/aspect_types/bigquery_table_config.json and change the aspect_type_project, aspect_type_region, and metadata_import_location to point to your Dataplex project, region and GCS bucket, respectively.

   ``` export IAM_TOKEN=$(gcloud auth print-identity-token)

   curl -X POST $TAG_ENGINE_URL/create_import_config -d @examples/configs/import/aspect_types/bigquery_table_config.json \ -H "Authorization: Bearer $IAM_TOKEN"

   curl -i -X POST $TAG_ENGINE_URL/trigger_job \ -d '{"config_type":"TAG_IMPORT","config_uuid":"23b772a24f7011efbbe242004e494300"}' \ -H "Authorization: Bearer $IAM_TOKEN"

   curl -X POST $TAG_ENGINE_URL/get_job_status -d '{"job_uuid":"23440e26501e11ef9de142004e494300"}' \ -H "Authorization: Bearer $IAM_TOKEN" ```

   Please note that you need to replace the config_uuid and job_uuid with your own values!

   The examples/configs/import/aspect_types folder also has working examples for creating bulk aspects on Fileset, Spanner, and Cloud SQL resources.

4. Run the following commands to create dynamic table-level aspects from BQ metadata:

   Open examples/configs/dynamic_table/aspect_types/dynamic_table_ondemand.json and change the aspect_type_project, aspect_type_region, and included_tables_uris to point to your Dataplex project, region, and BQ tables, respectively.

   ``` export IAM_TOKEN=$(gcloud auth print-identity-token)

   curl -X POST $TAG_ENGINE_URL/create_dynamic_table_config \ -d @examples/configs/dynamic_table/aspect_types/dynamic_table_ondemand.json \ -H "Authorization: Bearer $IAM_TOKEN"

   curl -i -X POST $TAG_ENGINE_URL/trigger_job \ -d '{"config_type":"DYNAMIC_TAG_TABLE","config_uuid":"59c51d0689ac11efb20b42004e494300"}' \ -H "Authorization: Bearer $IAM_TOKEN"

   curl -X POST $TAG_ENGINE_URL/get_job_status -d '{"job_uuid":"2679a3f66fa611efae5242004e494300"}' \ -H "Authorization: Bearer $IAM_TOKEN" ```

   Please note that you need to replace the config_uuid and job_uuid with your own values!

5. Run the following commands to create dynamic column-level aspects from BQ metadata:

   Open examples/configs/dynamic_column/aspect_types/dynamic_column_ondemand.json and change the aspect_type_project, aspect_type_region, included_columns_query, and included_tables_uris to point to your Dataplex project, region, BQ columns, and BQ tables, respectively.

   ``` export IAM_TOKEN=$(gcloud auth print-identity-token)

   curl -X POST $TAG_ENGINE_URL/create_dynamic_column_config \ -d @examples/configs/dynamic_table/aspect_types/dynamic_column_ondemand.json \ -H "Authorization: Bearer $IAM_TOKEN"

   curl -i -X POST $TAG_ENGINE_URL/trigger_job \ -d '{"config_type":"DYNAMIC_TAG_COLUMN","config_uuid":"9aaaed5089cf11ef825b42004e494300"}' \ -H "Authorization: Bearer $IAM_TOKEN"

   curl -X POST $TAG_ENGINE_URL/get_job_status -d '{"job_uuid":"b0a8e1de89cf11ef833d42004e494300"}' \ -H "Authorization: Bearer $IAM_TOKEN" ```

   Please note that you need to replace the config_uuid and job_uuid with your own values!

Part 3: Testing your Tag Engine API setup with Data Catalog API

1. Create the sample data_governance tag template:

   git clone https://github.com/GoogleCloudPlatform/datacatalog-templates.git cd datacatalog_templates python create_template.py $DATA_CATALOG_PROJECT $DATA_CATALOG_REGION tag_templates/data_governance.yaml

   The previous command creates the data_governance tag template in the $DATA_CATALOG_PROJECT and $DATA_CATALOG_REGION.

2. Grant permissions to invoker account (user or service):

   Depending on how you are involving the Tag Engine API, you'll need to grant permissions to either your service account or user account (or both).

   If you'll be invoking the Tag Engine API with a user account, authorize your user account as follows:

   ``` gcloud auth login

   export INVOKER_USER_ACCOUNT="username@example.com"

   gcloud iam service-accounts add-iam-policy-binding $TAG_CREATOR_SA \ --member=user:$INVOKER_USER_ACCOUNT --role=roles/iam.serviceAccountUser --project=$DATA_CATALOG_PROJECT

   gcloud run services add-iam-policy-binding tag-engine-api \ --member=user:$INVOKER_USER_ACCOUNT --role=roles/run.invoker \ --project=$TAG_ENGINE_PROJECT --region=$TAG_ENGINE_REGION ```

   If you are invoking the Tag Engine API with a service account, authorize your service account as follows:

   ``` export INVOKER_SERVICE_ACCOUNT="tag-engine-invoker@.iam.gserviceaccount.com"

   gcloud iam service-accounts add-iam-policy-binding $TAG_CREATOR_SA \ --member=serviceAccount:$INVOKER_SERVICE_ACCOUNT --role=roles/iam.serviceAccountUser

   gcloud run services add-iam-policy-binding tag-engine-api \ --member=serviceAccount:$INVOKER_SERVICE_ACCOUNT --role=roles/run.invoker \ --region=$TAG_ENGINE_REGION ```

   Very important: