GUID mapping

GUID mapping

The deploy API endpoint automatically generates and maintains a GUID mapping file in the config Git branch that records the GUID of the objects deployed from a Source Org (dev) to a Destination Org.

Note

ThoughtSpot never alters any GUIDs in the files stored in Git, regardless of branch. The find/replace of the mapping process occurs as the TML files in Git are imported by the /deploy REST API.

You can use the Git APIs across distinct ThoughtSpot instances, as long as they are configured to the same Git repository.

The GUID mapping files will be maintained as described in this article. If both instances don’t have Orgs enabled, the GUIDs may be identical. This is because, during object creation, ThoughtSpot will use a provided GUID in a TML file if that GUID doesn’t already exist on that instance.

GUID mapping files🔗

Each Org configured for Git integration will have a mapping JSON file generated with the name pattern org-{orgId}.json in the Git branch selected to host configuration files during the Git setup.

The mapping files for each ThoughtSpot instance will live in a directory in the config branch with name pattern {instanceId}.mapping. The instanceId is the id property returned from the REST API /system call.

Each mapping JSON file is an array of mapping objects recording the GUIDs for each TML object, which are automatically generated and maintained during each /deploy API call:

[
{
  "originalGuid" : "a5fc94bc-1d0f-4fa9-b7b6-7ce4dc6bd526",
  "mappedGuid" : "67804442-2568-4184-bbe4-2ee058e31431",
  "counter" : 0,
  "additionalMapping" : { }
},
{
  "originalGuid" : "cb04e13a-7969-42d5-9469-c3beb5182af6",
  "mappedGuid" : "91864ed5-63c2-4c9d-8d20-ba9a1e77a888",
  "counter" : 1,
  "additionalMapping" : {
    "bc46684e-d8f4-4a82-a38b-b6233329c1cd" : "c79fd844-5a1b-4645-87e7-2fb70f0f3421",
    "e0c0fb87-a9fe-43e1-93c6-f6a058711986" : "5872f7eb-24fc-4f4e-8f96-95afd24dc707",
    "941271b6-86fa-4777-a189-46a0e85d3917" : "09b90dc1-6f95-47b4-857e-b83ec98dda00",
  }
}
]

  • originalGuid refers to the GUID of the object on the source environment, typically the dev environment

  • mappedGuid refers to the GUID of the object on the destination environment, for example, release or prod

  • counter shows the number of times the mapped object was used in deploy operations.

  • additionalMapping is an object with direct key-value mappings of viz_ids on a Liveboard and is otherwise an empty object {}

The following figure illustrates how GUIDs are mapped during deployments:

GUID mapping

How mapping works🔗

The mapping file is updated when a new object is first created in the destination Org and receives its automatically generated GUID from the ThoughtSpot instance. Mapping is performed with an exact string Find/Replace operation, on the TML objects directly before the import, without modifying the files in the Git repository.

Using mapping for Table TML properties🔗

There are several properties within Table TML files related to a Table’s fully qualified name (FQN) within the RDBMS, that often need to be changed between Orgs.

You can use the exact string Find/Replace nature of the GUID Mapping file to define changes in the Table TMLs between environments.

Add the properties to the mapping file, including the TML property key along with the value, as shown here:

[
{
  "originalGuid" : "db: DEVDBNAME",
  "mappedGuid" : "db: PRODDBNAME",
  "counter" : 0,
  "additionalMapping" : { }
},
{
  "originalGuid" : "schema: DEV_SCHEMA",
  "mappedGuid" : "schema: CUSTOMER_A_SCHEMA",
  "counter" : 0,
  "additionalMapping" : { }
}
]

Note that the results will be an exact match on the provided string with exact string Find/Replace across the full set of TML deployed to that Org via Git.

If you need to limit the Find/Replace action to a specific TML file, you can instead put the rule within the additionalMapping section of the file:

{
  "originalGuid" : "cb04e13a-7969-42d5-9469-c3beb5182af6",
  "mappedGuid" : "91864ed5-63c2-4c9d-8d20-ba9a1e77a888",
  "counter" : 1,
  "additionalMapping" : {
    "db: DEVDBNAME" : "db: PRODDBNAME",
    "schema: DEV_SCHEMA" : "schema: CUSTOMER_A_SCHEMA"
  }
}

This requires more effort but works when a single Find/Replace against all files could result in incorrect results. Given there are often many Table TML files being redeployed, perhaps across different connections, additional specificity may be necessary.

Connection variations in each Org🔗

The Connection objects contain most details that vary between data environments, other than the db and schema properties within the table objects.

Connections can be defined programmatically via REST API or using TML for connections in the ThoughtSpot UI or via APIs.

Creating a connection involves secure credentials, which are not exported in the Connection TML. For this reason, it is preferable to set it via UI or REST API. An example of creating connections via REST API across multiple Orgs is available at the thoughtspot_rest_api_v1_python GitHub repository.

Switching "dev" Orgs🔗

When the Orgs feature is enabled on an instance for the first time, all existing content will be available in the Primary Org, with org_id as 0.

While the best practice is to create a separate dev Org and prod Org from the Primary Org, you may want to continue with Primary Org as prod Org for the time being and copy content to the dev Org to get started with the controlled SDLC processes.

You can accomplish this in phases using the Git APIs:

  1. Create a temporary Git branch and update the configuration to add Primary/prod Org to make it available to the commit operations.

  2. Use the Commit API to export all desired objects from the Primary/prod Org into the temporary Git branch (not the commit branch that you will use for deployment to the prod Org)

  3. Use the Deploy API into the new dev Org from the temporary branch to bring all the content into the dev Org. This will create the GUID mapping file for the dev Org.

  4. Take the JSON structure of the GUID mapping file from the dev Org, copy it, swap the keys and values, and then save the result in the GUID Mapping file of the prod.

Swap keys and values in the GUID Mapping file🔗

The structure of the GUID Mapping file is entirely simple key-value mappings in JSON, and thus can be easily swapped around programmatically.

You can modify the files in the config_branch manually for additions or corrections, but you must make sure the data is in the correct format and matches what ThoughtSpot generates.

An example of this process in Python is available as git_guid_mapping_file_functions.py.

Take the swapped JSON and save it in org-0.json file within the {instanceId}.mapping directory of the config_branch in GitHub, or whichever org-{orgId}.json file matches to your prod Org.

You can use pull requests to move content from dev to the prod branch, and when you push changes via /deploy API, the GUIDs will be swapped properly from their new origin in dev Org to the values in prod.