in Education by
I'm confused as to how I should use terraform to connect Athena to my Glue Catalog database. I use resource "aws_glue_catalog_database" "catalog_database" { name = "${var.glue_db_name}" } resource "aws_glue_crawler" "datalake_crawler" { database_name = "${var.glue_db_name}" name = "${var.crawler_name}" role = "${aws_iam_role.crawler_iam_role.name}" description = "${var.crawler_description}" table_prefix = "${var.table_prefix}" schedule = "${var.schedule}" s3_target { path = "s3://${var.data_bucket_name[0]}" } s3_target { path = "s3://${var.data_bucket_name[1]}" } } to create a Glue DB and the crawler to crawl an s3 bucket (here only two), but I don't know how I link the Athena query service to the Glue DB. In the terraform documentation for Athena, there doesn't appear to be a way to connect Athena to a Glue catalog but only to an S3 Bucket. Clearly, however, Athena can be integrated with Glue. How can I terraform an Athena database to use my Glue catalog as its data source rather than an S3 bucket? JavaScript questions and answers, JavaScript questions pdf, JavaScript question bank, JavaScript questions and answers pdf, mcq on JavaScript pdf, JavaScript questions and solutions, JavaScript mcq Test , Interview JavaScript questions, JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)

1 Answer

0 votes
by
Our current basic setup for having Glue crawl one S3 bucket and create/update a table in a Glue DB, which can then be queried in Athena, looks like this: Crawler role and role policy: The assume_role_policy of the IAM role needs only Glue as principal The IAM role policy allows actions for Glue, S3, and logs The Glue actions and resources can probably be narrowed down to the ones really needed The S3 actions are limited to those needed by the crawler resource "aws_iam_role" "glue_crawler_role" { name = "analytics_glue_crawler_role" assume_role_policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Principal": { "Service": "glue.amazonaws.com" }, "Effect": "Allow", "Sid": "" } ] } EOF } resource "aws_iam_role_policy" "glue_crawler_role_policy" { name = "analytics_glue_crawler_role_policy" role = "${aws_iam_role.glue_crawler_role.id}" policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:*", ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListBucket", "s3:GetBucketAcl", "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::analytics-product-data", "arn:aws:s3:::analytics-product-data/*", ] }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:*:*:/aws-glue/*" ] } ] } EOF } S3 Bucket, Glue Database and Crawler: resource "aws_s3_bucket" "product_bucket" { bucket = "analytics-product-data" acl = "private" } resource "aws_glue_catalog_database" "analytics_db" { name = "inventory-analytics-db" } resource "aws_glue_crawler" "product_crawler" { database_name = "${aws_glue_catalog_database.analytics_db.name}" name = "analytics-product-crawler" role = "${aws_iam_role.glue_crawler_role.arn}" schedule = "cron(0 0 * * ? *)" configuration = "{\"Version\": 1.0, \"CrawlerOutput\": { \"Partitions\": { \"AddOrUpdateBehavior\": \"InheritFromTable\" }, \"Tables\": {\"AddOrUpdateBehavior\": \"MergeNewColumns\" } } }" schema_change_policy { delete_behavior = "DELETE_FROM_DATABASE" } s3_target { path = "s3://${aws_s3_bucket.product_bucket.bucket}/products" } }

Related questions

0 votes
    I'm creating a list of IAM policy, which is stored in .json format. I have only 1 resource block ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 14, 2022 in Education by JackTerrance
0 votes
    I would like to avoid AWS dev endpoint. Is there a way where I can test and debug my PySpark ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 13, 2022 in Education by JackTerrance
0 votes
    Am trying to create a proxy for my hana db which will be called from a lamda function to write to ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 7, 2022 in Education by JackTerrance
0 votes
    Am trying to create a proxy for my hana db which will be called from a lamda function to write to ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 7, 2022 in Education by JackTerrance
0 votes
    The Cookies Preferences box that always appears in the bottom right hand corner of the dataplatform.ibm.com ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 23, 2022 in Education by JackTerrance
0 votes
    I am creating a below ABC -- ecs.tf (gives me cluster id) Content of ecs.tf: resource " ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Apr 6, 2022 in Education by JackTerrance
0 votes
    I'm experiencing some problems regarding Twitter OAuth within an android activity. I read a lot of tutorials ... Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Feb 27, 2022 in Education by JackTerrance
0 votes
    What is the purpose of a glue language? (a) Format data (b) Product data (c) Collect data (d) ... , JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Oct 23, 2021 in Education by JackTerrance
0 votes
    Every time I try to run a small application that uses a Derby DB I get this error message: ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Mar 23, 2022 in Education by JackTerrance
0 votes
    Every time I try to run a small application that uses a Derby DB I get this error message: ... JavaScript Questions for Interview, JavaScript MCQ (Multiple Choice Questions)...
asked Mar 23, 2022 in Education by JackTerrance
0 votes
    Is there any way to stop a SQL Azure DB so that it doesn't charge anything towards our account? I don't ... cloud services and VM's. Select the correct answer from above options...
asked Jan 31, 2022 in Education by JackTerrance
0 votes
    __________ contains a huge proxy DB which users can use to protect their online privacy and preserve their ... -Cyber Security:,Cyber Security-Jobs:,Cyber Security Applications...
asked Oct 31, 2021 in Education by JackTerrance
0 votes
    Find the command that forces MongoDB to use a particular index for a db.collection.find() operation. (1)explain() (2)hint() (3)hit()...
asked May 22, 2021 in Technology by JackTerrance
0 votes
    I've got two different apps that I am hosting (well the second one is about to go up) on Amazon EC2. ... path ; ec2-describe-instances Select the correct answer from above options...
asked Feb 4, 2022 in Education by JackTerrance
0 votes
    I'm having trouble using * in the AWS CLI to select a subset of files from a certain bucket. Adding * to the ... s3://data/2016-08*. Select the correct answer from above options...
asked Jan 30, 2022 in Education by JackTerrance
...