Create an Object-Backed Data Source

Amazon S3

1. Select an Authentication Method. You must choose an authentication method before connecting an S3 data source. The following methods are supported:

   • AWS Access Key: Connect to a private S3 bucket using an access key pair.
   • AWS Instance Role: Connect to a private S3 bucket with no credentials, instead leveraging an IAM Role that has been assigned to the Immuta EC2 instance. The Data Owner must possess the CREATE_S3_DATASOURCE_WITH_INSTANCE_ROLE permission to proceed. By default, this option is disabled. For instructions on enabling this option, please contact your Immuta Support Professional.
   • No Authentication: Connect to a public S3 bucket with no credentials.

   

2. Fill out the following fields in the Connection Information window:

   • AWS Access Key Id: Your AWS public access key.
   • AWS Secret Access Key: Your AWS secret access key.
   • Assume AWS IAM Role: Enable only if you want to connect using an IAM Role's access as opposed to an individual user's access.
   • AWS IAM Role ARN: If Assume AWS IAM Role is enabled, this is the name of the IAM Role ARN that will be used (please login to the AWS Console or contact your AWS Admin for this information).
   • AWS Region: The region that contains the S3 bucket that you wish to expose.
   IAM Policy Elements

   To connect to an S3 bucket, the IAM role you use must have several policy elements. These elements are included in a Statement array and must be present for every bucket you wish to connect with that IAM Role.

   • s3:ListBucket on the bucket you want to expose.
   • s3:getObject on /* of the bucket you want to expose.
   • s3:getObjectTagging on /* of the bucket you want to expose.

   Optionally, you can give the element s3:ListAllMyBuckets on arn:aws:s3:::*. This allows Immuta to present you with a list of buckets to choose from at data source creation time. Without this, you'll have to type in your bucket name.

   In order to update metadata from S3 as changes occur, additional policy elements are required.

   Below is an example policy that can be used with the buckets test-bucket-1 and test-bucket-2. Note, that you'd have to create a data source per bucket, but that this policy (applied to your IAM role) is applicable to both buckets. This policy does not include the list.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::test-bucket-1",
                   "arn:aws:s3:::test-bucket-2"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:getObject",
                   "s3:getObjectTagging"
               ],
               "Resource": [
                   "arn:aws:s3:::test-bucket-1/*",
                   "arn:aws:s3:::test-bucket-2/*"
               ]
           }
       ]
   }
   

3. Click Verify Credentials.

4. Complete the following fields in the S3 Configuration window:

   • S3 Bucket: the bucket you want your data source to reference
   • S3 Prefix: the prefix within the specified bucket. Only blobs from within the specified prefix will be ingested and returned.
   • Additional Options:
     • Ingest Metadata Automatically: when enabled, this automatically monitors S3 for new updates to the data. If not enabled, there will only be a one-time metadata pull from S3. No additions or deletions will be captured after the pull completes. That means even if you change S3 attributes, those changes will not be reflected in the data source policy. This feature creates SQS queues containing bucket updates. Note that even if you do not enable this option, you can still direct Immuta to re-crawl the data manually whenever you would like.

   

5. If you select Ingest Metadata Automatically, enter the name of the SQS queue in the SQS Queue field.

   These AWS IAM permissions are required:

   • sqs:DeleteMessageBatch a general permission on SQS.
   • sqs:ReceiveMessage a general permission on SQS.

Advanced Options

In this section, you can edit advanced configurations for your data source. None of these configurations are required to create the data source.

Azure Blob Storage

1. To connect a data source to an Azure Blob Storage container, you must first create a Shared Access Signature for your Azure Blob Storage account.

2. Enter the Shared Access Signature Token and the corresponding URL for your Azure Storage account. Follow the steps below to retrieve your SAS credentials from the Azure Portal.

   

3. Open the Azure Portal Web UI.

4. Find and select your desired Azure Storage Account resource.

5. Under SETTINGS select Shared access signature.

   

6. Configure the SAS Token's allowed services, resource types, and permissions to match the following image.

   

7. Set a reasonable expiration date for your SAS Token. When your SAS Token expires, your Immuta data source will no longer be able to fetch data from Azure.

8. Select Generate SAS and save the provided credentials.

9. Select the container that you wish to base this data source on. The data source will contain all of the blobs in this container, and it will also maintain the container's directory structure.

10. On the data source creation page in the Immuta console, enter the Blob Storage SAS Token and Token URL and click Verify Credentials.

11. Enter the Azure Blob Storage Container.

Advanced Options

In this section, you can edit advanced configurations for your data source. None of these configurations are required to create the data source.