How To Setup Data Pipeline - Destination Setup

Azure Setup Guide

Pre-requisites:

• Brands should have an Azure account with ADLS (Gen2) enabled storage accounts
• For the purpose of this setup, Microsoft Azure Storage Explorer is used in the documentation. Other relevant platform/ tool can also be used for the same

Steps Involved:

• Open Microsoft Azure Storage Explorer and navigate to Storage Accounts > {Account Name} > Blob containers > {Container Name}
  

• Right click on {container name} > Select “Get Shared Access Signature” - SAS Token
  

• Specify the SAS token start and expiry range and associated permission:

  • Create / Write / Delete are mandatory permission required while Read will be required for investigational purposes
    

• Post Creation, Copy the SAS URL and upload in the Data Pipeline config as mentioned in the Data Pipeline Setup documentation:
  

AWS (S3) Setup Guide

Pre-requisites:

• PAR Ordering platform AWS access role should be allowed in the S3 bucket policy to put the object.

• Source AWS account root user should be allowed in key policy to encrypt the objects. (If KMS is required).

Essential requirements for setup:

1. S3 Bucket AWS region, S3 bucket ARN, and the sub-folder name

2. KMS key AWS region and KMS key ARN (S3 bucket and KMS should be in the same region - if KMS encryption is required)
   

Steps Involved:

There are a few policy additions required to set targets for the data pipeline. Given below AWS resources need to have permission:

1. The customer needs to add server-side encryption - Optional

2. S3: Bucket policy will be added to target bucket with given policy and authorize PAR Ordering AWS role to perform few actions on the bucket. Required permissions are "s3:PutObject", "s3:DeleteObject", "s3:PutObjectAcl", "s3:ListBucket".
   For example:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "dmscrossaccountupload",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::653800670028:role/par-MENU-platform-datapipeline"

},
"Action": [
"s3:PutObject",
"s3:DeleteObject",
"s3:PutObjectAcl",
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::<Customer Bucket Name>/*",
"arn:aws:s3:::<Customer Bucket Name>"
]
}
]
}

1. KMS: Key policy will be added ( If KMS encryption required) to the dedicated KMS key where below given permission are required to PAR Ordering AWS role to perform action for encryption. The required permission are "kms:Encrypt","kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*","kms:DescribeKey""kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"

{
 "Sid": "Allow use of the key",
 "Effect": "Allow",
 "Principal": {
 "AWS": [
 "arn:aws:iam::653800670028:root",
 "arn:aws:iam::653800670028:role/par-MENU-platform-datapipeline"
 ]
 "Action": [
 "kms:Encrypt",
 "kms:Decrypt",
 "kms:ReEncrypt*",
 "kms:GenerateDataKey*",
 "kms:DescribeKey"
 ],
 "Resource": "*"
 },
 {
 "Sid": "Allow attachment of persistent resources",
 "Effect": "Allow",
 "Principal": {
 "AWS": [
 "arn:aws:iam::653800670028:root",
 "arn:aws:iam::653800670028:role/par-MENU-platform-datapipeline"
 ]
 },
 "Action": [
 "kms:CreateGrant",
 "kms:ListGrants",
 "kms:RevokeGrant"
 ],
 "Resource": "*",
 "Condition": {
 "Bool": {
 "kms:GrantIsForAWSResource": "true"
 }
 }
 }

The S3 bucket complete policy will look like if we use KMS key:

{
 "Version": "2012-10-17",
 "Id": "key-consolepolicy-3",
 "Statement": [
 {
 "Sid": "Enable IAM User Permissions",
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::<target AWS A/C ID>:root"
 },
 "Action": "kms:*",
 "Resource": "*"
 },
 {
 "Sid": "Allow access for Key Administrators",
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::<target AWS A/C ID>:root"
 },
 "Action": [
 "kms:Create*",
 "kms:Describe*",
 "kms:Enable*",
 "kms:List*",
 "kms:Put*",
 "kms:Update*",
 "kms:Revoke*",
 "kms:Disable*",
 "kms:Get*",
 "kms:Delete*",
 "kms:TagResource",
 "kms:UntagResource",
 "kms:ScheduleKeyDeletion",
 "kms:CancelKeyDeletion"
 ],
 "Resource": "*"
 },
 {
 "Sid": "Allow use of the key",
 "Effect": "Allow",
 "Principal": {
 "AWS": [
 "arn:aws:iam::653800670028:root",
 "arn:aws:iam::653800670028:role/par-MENU-platform-datapipeline"
 ]
 },
 "Action": [
 "kms:Encrypt",
 "kms:Decrypt",
 "kms:ReEncrypt*",
 "kms:GenerateDataKey*",
 "kms:DescribeKey"
 ],
 "Resource": "*"
 },
 {
 "Sid": "Allow attachment of persistent resources",
 "Effect": "Allow",
 "Principal": {
 "AWS": [
 "arn:aws:iam::653800670028:root",
 "arn:aws:iam::653800670028:role/par-MENU-platform-datapipeline"
 ]
 },
 "Action": [
 "kms:CreateGrant",
 "kms:ListGrants",
 "kms:RevokeGrant"
 ],
 "Resource": "*",
 "Condition": {
 "Bool": {
 "kms:GrantIsForAWSResource": "true"
 }
 }
 }
 ]
}

Please note: The above policy is specific to the KMS key. The authorized action will be applicable only on Specific keys.