Skip to content

You are viewing documentation for Immuta version 2022.5.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

cancel

Create and Apply a Template to a Data Source

Audience: Data Governors and Data Owners

Content Summary: Although only Data Governors can create classifiers, Data Owners can add classifiers to templates, which they then apply to their data sources to explicitly apply tags defined in classifiers within the template. Templates are created by Data Owners and may be used (or cloned) by other Data Owners; however, with the exception of Data Governors, only the template creator can modify or delete it. Additionally, a template can only be deleted if it is not currently applied to any data sources. This page outlines how to create a template and apply it to a data source.

Additional Tutorials:

1 - Create a Template

  1. Generate your API key on the API Keys tab on your profile page and save the API key somewhere secure. You will include this API key in the authorization header when you make a request to the Immuta API.

  2. Find classifiers to include in your template using one of these methods:

    Immuta CLI

    immuta api sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=CLASSIFIER

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=CLASSIFIER

  3. If the request was successful, you will receive a list of available classifiers.

    {
  "count": 3,
  "hits": [
    {
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "name": "ACCOUNT_NUMBER_CLASSIFIER",
      "displayName": "Account Number Classifier",
      "description": "This classifier identifies account numbers using a regex",
      "type": "regex",
      "config": {
        "tags": [
          "Discovered.account-number"
        ],
        "regex": "[0-9]{9}-[0-9]{3}-[0-9]{1}",
        "minConfidence": 0.5
      },
      "id": 104,
      "createdAt": "2021-10-20T19:12:24.889Z",
      "updatedAt": "2021-10-20T19:12:24.889Z"
    },
    {
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "name": "EMPLOYEE_DESK_LOCATION_CLASSIFIER",
      "displayName": "Employee Desk Location Classifier",
      "description": "This classifier detects when an employee's desk location appears in a dataset.",
      "type": "dictionary",
      "config": {
        "tags": [
          "Discovered.desk-location"
        ],
        "values": [
          "Research Lab",
          "Blue Room",
          "Purple Room"
        ],
        "caseSensitive": false,
        "minConfidence": 0.6
      },
      "id": 68,
      "createdAt": "2021-10-20T17:57:51.696Z",
      "updatedAt": "2021-10-20T17:57:51.696Z"
    },
    {
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_CLASSIFIER",
      "displayName": "Social Security Number Columns Classifier",
      "description": "This classifier identifies column names that match the defined regex pattern.",
      "type": "columnNameRegex",
      "config": {
        "tags": [
          "Discovered.Social Security Numbers"
        ],
        "columnNameRegex": "ssn|social ?security"
      },
      "id": 67,
      "createdAt": "2021-10-20T17:57:17.930Z",
      "updatedAt": "2021-10-20T17:57:17.930Z"
    }
  ]
}

  4. Save the template payload in a .json file. Use the tabs below to see different examples of templates.

    {
  "name": "ACCOUNT_NUMBERS_TEMPLATE",
  "displayName": "Account Numbers Template",
  "description": "This template contains the classifier that identifies account numbers.",
  "classifiers": [
    {
      "name": "ACCOUNT_NUMBER_CLASSIFIER"
    }
  ],
  "sampleSize": 100
}

    {
  "name": "EMPLOYEE_DESK_LOCATION_TEMPLATE",
  "displayName": "Employee Desk Location Template",
  "description": "This template contains the classifier that detects when the name of the room an employee's desk is in appears in a dataset.",
  "classifiers": [
    {
      "name": "EMPLOYEE_DESK_LOCATION_CLASSIFIER"
    }
  ],
  "sampleSize": 100
}

    {
  "name": "SOCIAL_SECURITY_NUMBERS_TEMPLATE",
  "displayName": "Social Security Numbers Template",
  "description": "This template contains the classifier that matches social security number column names with the defined regex.",
  "classifiers": [
    {
      "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_CLASSIFIER"
    }
  ],
  "sampleSize": 100
}

    {
  "name": "STUDENT_LOCATION_TEMPLATE",
  "displayName": "Student Location Template",
  "description": "This template contains the classifier that detects when a student's residence hall, floor, or room appears in a dataset.",
  "classifiers": [
    {
      "name": "STUDENT_LOCATION_CLASSIFIER"
    }
  ],
  "sampleSize": 100
}

  5. Create the template:

    Immuta CLI

    immuta api sdd/template -X POST --input ./example-payload.json

    HTTP API

    curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template

  6. If the request is successful, you will receive a response that contains details about the template. Use the tabs below to see different responses for different templates.

    {
  "name": "ACCOUNT_NUMBERS_TEMPLATE",
  "displayName": "Account Numbers Template",
  "description": "This template contains the classifier that identifies account numbers.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T19:12:22.092Z",
  "updatedAt": "2021-10-21T19:12:22.092Z",
  "classifiers": [
    {
      "name": "ACCOUNT_NUMBER_CLASSIFIER",
      "overrides": {}
    }
  ]
}

    After the template is applied to data sources and Sensitive Data Discovery is run, the Discovered.account-number tag will be applied to columns that Immuta identifies with 50% confidence, as configured in the classifier.

    {
  "name": "EMPLOYEE_DESK_LOCATION_TEMPLATE",
  "displayName": "Employee Desk Location Template",
  "description": "This template contains the classifier that detects when the name of the room an employee's desk is in appears in a dataset.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T18:03:58.967Z",
  "updatedAt": "2021-10-21T18:03:58.967Z",
  "classifiers": [{
    "name": "EMPLOYEE_DESK_LOCATION_CLASSIFIER",
    "overrides": {}
  }]
}

    After the template is applied to data sources and Sensitive Data Discovery is run, the Discovered.desk-location tag will be applied to columns when Immuta detects the values Research Lab, Blue Room or Purple Room with 60% confidence, as configured in the classifier.

    {
  "name": "SOCIAL_SECURITY_NUMBERS_TEMPLATE",
  "displayName": "Social Security Numbers Template",
  "description": "This template contains the classifier that matches social security number column names with the defined regex.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 2,
  "createdAt": "2021-10-21T19:12:22.092Z",
  "updatedAt": "2021-10-21T19:12:22.092Z",
  "classifiers": [
    {
      "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_CLASSIFIER",
      "overrides": {}
    }
  ]
}

    After the template is applied to data sources and Sensitive Data Discovery is run, the Discovered.social-security-number tag will be applied to columns that have a name that match the ssn|social ?security regex, such as ssn, socialsecurity, or social security.

    {
  "name": "STUDENT_LOCATION_TEMPLATE",
  "displayName": "Student Location Template",
  "description": "This template contains the classifier that detects when a student's residence hall, floor, or room appears in a dataset.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T18:03:58.967Z",
  "updatedAt": "2021-10-21T18:03:58.967Z",
  "classifiers": [{
    "name": "STUDENT_LOCATION_CLASSIFIER",
    "overrides": {}
  }]
}

    After the template is applied to data sources and Sensitive Data Discovery is run, the Discovered.residence-hall tag will be applied to columns when Immuta detects values that match those listed in the Residence Halls data source with 70% confidence, as configured in the classifier.

2 - Apply a Template to Data Sources

Attributes of all custom classifiers and templates are provided on the Sensitive Data Discovery API page. However, attributes specific to this section are outlined in the table below.

Attribute Description
template string The name of the template to apply to the data sources; null clears the current template.
sources string The name of the data sources to apply the template to.

  1. Find templates to apply to your data sources:

    Immuta CLI

    immuta api sdd/template

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/template

  2. If the request was successful, you will receive a list of available templates.

    {
  "count": 3,
  "hits": [
    {
      "name": "ACCOUNT_NUMBERS_TEMPLATE",
      "displayName": "Account Numbers Template",
      "description": "This template contains the classifier that identifies account numbers.",
      "sampleSize": 100,
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "id": 2,
      "createdAt": "2021-10-20T19:13:35.319Z",
      "updatedAt": "2021-10-20T19:13:35.319Z",
      "classifiers": [
        {
          "name": "ACCOUNT_NUMBER_CLASSIFIER",
          "overrides": {}
        }
      ]
    },
    {
      "name": "EMPLOYEE_DESK_LOCATION_TEMPLATE",
      "displayName": "Employee Desk Location Template",
      "description": "Contains classifier that detects when the name of a room a desk is in appears in a dataset.",
      "sampleSize": 100,
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "id": 1,
      "createdAt": "2021-10-20T18:03:58.967Z",
      "updatedAt": "2021-10-20T18:03:58.967Z",
      "classifiers": [
        {
          "name": "EMPLOYEE_DESK_LOCATION_CLASSIFIER",
          "overrides": {}
        }
      ]
    },
    {
      "name": "SOCIAL_SECURITY_NUMBERS_TEMPLATE",
      "displayName": "Social Security Numbers Template",
      "description": "Contains classifier that matches ssn column names with the defined regex.",
      "sampleSize": 100,
      "createdBy": {
        "id": 1,
        "name": "John",
        "email": "john@example.com"
      },
      "id": 3,
      "createdAt": "2021-10-20T19:13:58.359Z",
      "updatedAt": "2021-10-20T19:13:58.359Z",
      "classifiers": [
        {
          "name": "SOCIAL_SECURITY_NUMBER_COLUMNS_CLASSIFIER",
          "overrides": {}
        }
      ]
    }
  ]
}

  3. Select an appropriate template to apply to your data sources, and save the payload in a .json file:

    {
  "template": "ACCOUNT_NUMBERS_TEMPLATE",
  "sources": [
    "Insurance Data"
  ]
}

  4. Apply the template to your data source(s):

    Immuta CLI

    immuta api sdd/template/apply -X PUT --input ./example-payload.json

    HTTP API

    curl \
    --request PUT \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer dea464c07bd07300095caa8" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template/apply

  5. You will receive a response that indicates whether or not the template was successfully applied to your data sources.

    {
  "success": true
}

Additional Tutorials

Clone a Template

Users cannot modify templates created by other Data Owners, but they can clone templates and make changes to the clone.

  1. Get a list of templates to determine the template you want to clone using one of these methods:

    Immuta CLI

    immuta api sdd/sdd/template?sortField=name&sortOrder=asc&offset=5&limit=5

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/template?sortField=name&sortOrder=asc&offset=5&limit=5

  2. Save the template clone name and details in a .json file.

    {
  "name": "INSURANCE_ACCOUNT_NUMBERS",
  "displayName": "Insurance Account Numbers",
  "description": "This template is specific to insurance accounts."
}

  3. Clone the template:

    Immuta CLI

    immuta api sdd/template/ACCOUNT_NUMBERS_TEMPLATE/clone -X POST --input ./example-payload.json

    HTTP API

    curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template/ACCOUNT_NUMBERS_TEMPLATE/clone

  4. If the request was successful, you will receive a response that provides details about the template clone.

    {
  "name": "INSURANCE_ACCOUNT_NUMBERS",
  "displayName": "Insurance Account Numbers",
  "description": "This template is specific to insurance accounts.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 4,
  "createdAt": "2021-10-20T20:48:37.805Z",
  "updatedAt": "2021-10-20T20:48:37.805Z",
  "classifiers": [
    {
      "name": "ACCOUNT_NUMBER_CLASSIFIER",
      "overrides": {}
    }
  ]
}

You can now modify the template, such as changing the classifiers included and the sampleSize.

Configure Classifier Tags and Confidence

To disable classifier tags from being set, you can create a template to that configures the classifier that contains that tag.

For example, the built-in PERSON_NAME classifier contains the following tags: Discovered.PHI, Discovered.PII, Discovered.Entity.Person Name, and Discovered.Identifier Indirect. However, your organization doesn't have any health data, so you don't want the PHI tag to be applied to your data sources but you do want all the other tags within that classifier.

To override the Discovered.PHI tag, you would create a template that includes the PERSON_NAME classifier and removes the Discovered.PHI from the list of tags in the template payload.

  1. View the details about the PERSON_NAME classifier so you know what to include in your template using one of these methods:

    Immuta CLI

    immuta api sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=PERSON_NAME

    HTTP API

    curl \
    --request GET \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    https://your-immuta-url.immuta.com/sdd/classifier?sortField=name&sortOrder=asc&limit=25&searchText=PERSON_NAME

  2. If the request was successful, the response will include details about the PERSON_NAME classifier.

    {
  "createdBy": {
    "id": 21,
    "name": "Immuta System Account",
    "email": "immuta_system@immuta.com"
  },
  "name": "PERSON_NAME",
  "displayName": "Person Name",
  "description": "Detects strings consistent with a dictionary of people's names.",
  "type": "builtIn",
  "config": {
    "tags": [
      "Discovered.PHI",
      "Discovered.PII",
      "Discovered.Entity.Person Name",
      "Discovered.Identifier Indirect"
    ],
    "minConfidence": 0.3
  },
  "id": 54,
  "createdAt": "2021-10-21T07:35:14.416Z",
  "updatedAt": "2021-10-21T12:57:43.919Z"
}

  3. Remove the Discovered.PHI tag from the list of tags in the classifier config, and save the template payload in a .json file.

    {
  "name": "PERSON_NAME_OVERRIDE",
  "displayName": "Person Name Override",
  "description": "This template removes the PHI tag from the PERSON_NAME classifier.",
  "classifiers": [
    {
      "name": "PERSON_NAME",
        "overrides": {
          "tags": [
            "Discovered.PII",
            "Discovered.Entity.Person Name",
            "Discovered.Identifier Indirect"
          ]
        }
      }
    ],
  "sampleSize": 100
}

  4. Create the template:

    Immuta CLI

    immuta api sdd/template -X POST --input ./example-payload.json

    HTTP API

    curl \
    --request POST \
    --header "Content-Type: application/json" \
    --header "Authorization: 12345678900000" \
    --data @example-payload.json \
    https://your-immuta-url.immuta.com/sdd/template

If the request is successful, you will receive a response that details the new template:

{
  "name": "PERSON_NAME_OVERRIDE",
  "displayName": "Person Name Override",
  "description": "This template removes the PHI tag from the PERSON_NAME classifier.",
  "sampleSize": 100,
  "createdBy": {
    "id": 1,
    "name": "John",
    "email": "john@example.com"
  },
  "id": 1,
  "createdAt": "2021-10-21T17:11:18.057Z",
  "updatedAt": "2021-10-21T17:11:18.057Z",
  "classifiers": [
    {
      "name": "PERSON_NAME",
      "overrides": {
        "tags": [
          "Discovered.PII",
          "Discovered.Entity.Person Name",
          "Discovered.Identifier Indirect"
        ]
      }
    }
  ]
}

What's Next

Now that you've created a template, continue to one of the following tutorials: