Today Whit Dabbles in AWS Cloud Formation. If you recall from previous posts, I am a Data Engineer in the Healthcare SaaS (software as a service) space. One of the tasks I have been asked to complete recently is to architect a Data Lake for our many application stores. The stores range from PostgreSQL to Aurora to MySQL to RDS....you name it, we probably have it. Architecting the lake started with our primary product's data store, housed in both standalone MySQL and Aurora MySQL variants, across a multi-tenant CRM environment. Being an AWS Cloud house, a lot of the work we do is engineered for redeployment across accounts using infrastructure as code. There are two flavors of this type of coding, standard Cloud Formation and SAM, or serverless application model. SAM is an extension of Cloud Formation itself and makes it way easier to deploy features such as Lambdas and other serverless components. Here is a diagram for the initial, standard Cloud Formation for our multi-tenant sources.
After architecting the basics of the data lake and completing about 50% of the above flow, we were asked to migrate two tables from a new microservice, housed in PostgreSQL, to Snowflake, our Cloud Data Warehouse provider. Snowflake is an awesome columnar data store with features to connect directly to your cloud provider of choice, but I will save that discussion for another day. The main reason I wanted to bring up this process is that I got HELLA stuck on a piece of the Cloud Formation necessary for the data flow.
This flow utilizes the SAM extension for Cloud Formation because most of the operations are serverless. What had me stuck was after creating the bucket and the queue, both encrypted via a KMS CMK (Customer Managed Key) and deploying the Cloud Formation, I kept being met with a nondescript error that read, "Unable to validate the following destination configurations....S3..." As it turns out, a policy is necessary to allow S3 to send messages to the queue.
MyQueuePolicy:
Type: AWS::SQS::QueuePolicy
DependsOn:
- MyQueue
Properties:
Queues:
- !Ref MyQueue
PolicyDocument:
Id: !Join ['-', [!Ref EnvironmentLower, "my-stack-name"]]
Statement:
- Sid: !Join ['-', [!Ref EnvironmentLower, "my-stack-name-inner-policy"]]
Effect: Allow
Principal: "*"
Action: "SQS:SendMessage"
Resource: !GetAtt MyQueue.Arn
Condition:
ArnLike:
aws:SourceArn: !Join ['', ["arn:aws:s3:::", !Ref MyBucketName]]Another thing that is important is the order in which you are creating the components. What I didn't realize at the time, is that the generic error I was seeing can be thrown for both missing permissions and circular dependencies. A circular dependency prohibits one item from creating because it is looking for another, and that item is looking for the first item as well. Crazy, right?! Because the Bucket references the SQS queue, the queue needs to be created in sequence ahead of the bucket. You can do this with references (!Ref) or via DependsOn clauses in the code. As you can see above, the queue policy has a DependsOn clause in order to force it to wait for the creation of the queue. It makes the most sense to me to create in the order, Queue > Policy > Bucket. Because of this, I utilized DependsOn for my Bucket as well, citing both the queue and the policy (I will paste the entirety of the code snippet below). That should get things going in the right order at least.
Awesome! Now that we have that figured out, let's deploy again--bummer, same failure! The last piece that escaped me was that the KMS key needs to allow the principal, the s3 service, to utilize the key as part of keeping the data encrypted while in flight. Keep in mind that you should limit your resource to only the necessary items, but this will at least get you going
{
"Sid": "testing",
"Effect": "Allow",
"Principal": {
"Service": "s3.amazonaws.com"
},
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt"
],
"Resource": "*"
}Once I added this to the policy for the KMS key I was golden--deployed my code without issue, all resources created perfectly! This victory was a long time coming and I never could find a good resource online for how to enable the data flow where KMS was employed. There are quite a few stack overflow conversations regarding the movement without that piece, but we all know data has to stay safe--encrypt everything!
If you've made it this far you'll be one step ahead of those who simply skip to copy-pasta the code--you're awesome! Here is how the code looks in YAML flavor for just the resources we discussed here. Keep in mind that the DLQ code is not included but if you have questions please feel free to ask. I'd be happy to work through any issues with you and HAPPY CODING!
Resources:
MyBucket:
Type: 'AWS::S3::Bucket'
DependsOn:
- MyQueuePolicy
- MyQueue
Properties:
BucketName: !Ref MyBucket
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: 'aws:kms'
KMSMasterKeyID: !Sub ${KMSKeyId}
NotificationConfiguration:
QueueConfigurations:
- Event: "s3:ObjectCreated:*"
Queue: !Join ['', ["arn:aws:sqs:us-west-2:", !Ref AWS::AccountId, ":", !Ref EnvironmentLower, "-my-stack-name"]]
Tags:
- Key: Environment
Value: !Sub ${Environment}
MyQueuePolicy:
Type: AWS::SQS::QueuePolicy
DependsOn:
- MyQueue
Properties:
Queues:
- !Ref MyQueue
PolicyDocument:
Id: !Join ['-', [!Ref EnvironmentLower, "my-stack-name"]]
Statement:
- Sid: !Join ['-', [!Ref EnvironmentLower, "my-stack-name-inner-policy"]]
Effect: Allow
Principal: "*"
Action: "SQS:SendMessage"
Resource: !GetAtt MyQueue.Arn
Condition:
ArnLike:
aws:SourceArn: !Join ['', ["arn:aws:s3:::", !Ref MyBucket]]
MyQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Join ['-', [!Ref EnvironmentLower, "my-stack-name"]]
MessageRetentionPeriod: 86400
VisibilityTimeout: 3600
KmsMasterKeyId: !Ref KMSKeyId
RedrivePolicy:
deadLetterTargetArn: !GetAtt MyDLQ.Arn
maxReceiveCount: 3
Tags:
- Key: Name
Value: !Join ['-', [!Ref EnvironmentLower, "my-stack-name"]]
Comentarios