Simplify enterprise knowledge entry utilizing the Amazon Redshift integration with Amazon S3 Entry Grants

June 1, 2025

2

Scaling knowledge entry securely whereas sustaining operational effectivity is a crucial problem for organizations. Entry rights are sometimes fragmented throughout numerous AWS companies, as totally different enterprise items personal and handle totally different knowledge shops, similar to Amazon Easy Storage Service (Amazon S3) and Amazon Redshift. As knowledge grows, modeling entry in AWS Id and Entry Administration (IAM) insurance policies turns into difficult for knowledge house owners, as they attempt to handle entry for various teams and customers throughout accounts within the group. Managing these distributed entry rights requires substantial overhead, as a result of safety groups and knowledge house owners should collaborate to replace and monitor permissions to verify knowledge is simply accessible to licensed customers.

Recognizing this problem, the Amazon S3 Entry Grants integration with Amazon Redshift permits centralized consumer authentication by way of AWS IAM Id Middle, offering unified id throughout the group. S3 Entry Grants permits particular IAM Id Middle customers or teams to entry registered Amazon S3 places by way of a grant. Making a grant with a gaggle as grantee lets the group members entry solely the S3 bucket, prefix, or object inside the grant’s scope. Which means that entry might be managed by merely making a grant for a gaggle and including or eradicating the consumer from the group, decreasing administrative overhead.

On this put up, we present the best way to grant Amazon S3 permissions to IAM Id Middle customers and teams utilizing S3 Entry Grants. We additionally take a look at the mixing utilizing an IAM Id Middle federated consumer to unload knowledge from Amazon Redshift to Amazon S3 and cargo knowledge from Amazon S3 to Amazon Redshift.

Resolution overview

This put up covers a use case the place a big group manages 1000’s of company customers throughout a number of enterprise items by way of their id supplier (IdP). These customers usually work together with huge quantities of information saved throughout quite a few S3 buckets, regularly performing extract, rework, and cargo (ETL) operations by way of Amazon Redshift. Their aim is to have an easier ETL course of of information loading and unloading operations in Amazon Redshift with out managing a number of IAM roles and insurance policies for Amazon S3 entry. Additionally, they need a centralized entry administration resolution that seamlessly integrates their company identities from present IdP with AWS companies.

For this resolution, AWS Organizations is enabled and IAM Id Middle is configured within the delegated administration account. The group has two member accounts: Member Account 1 runs analytical workloads on Amazon Redshift, with all of the companies enabled with trusted id propagation, and Member Account 2 manages knowledge saved in Amazon S3; right here you’ll arrange S3 Entry Grants. Amazon Redshift will load the user-specific knowledge from Amazon S3 saved in Member Account 2 utilizing entry management primarily based on IAM Id Middle customers and teams. This improves the consumer expertise sustaining a single authentication mechanism inside a corporation, retaining entry management, and useful resource separation utilizing AWS accounts as a boundary per enterprise items.

The next diagram illustrates the answer structure.

Figure 1: Architecture showing the solution

Determine 1: Structure exhibiting the answer

To run this resolution in a single account, configure Amazon Redshift and S3 Entry Grants with account cases of IAM Id Middle. Assessment When to make use of account cases for extra data.

The answer workflow contains the next steps:

The consumer configures and connects with their respective purchasers (similar to Amazon Redshift Question Editor v2 or a SQL consumer) to entry Amazon Redshift utilizing IAM Id Middle.
A brand new browser home windows opens and is redirected to the login web page of the IdP.
The consumer logs in with their IdP consumer title and password.
After the login is profitable, the consumer is redirected to the consumer software, such because the Amazon Redshift Question Editor.
When the consumer tries to entry knowledge in Amazon S3 utilizing the LOAD or UNLOAD SQL command, Amazon Redshift in Member Account 1 will request credentials from the S3 Entry Grants occasion from Member Account 2, the place the Amazon S3 knowledge is saved. This request will comprise the consumer context.
S3 Entry Grants will then consider the request in opposition to the grants it has, matching the id specified within the grant with the one obtained within the request. If there’s a match, the requestor will obtain momentary entry to the Amazon S3 places specified within the grant’s scope.

To implement the answer, we stroll you thru the next steps:

Allow S3 Entry Grants in your Amazon Redshift managed software.
Replace IAM function permissions used within the software.
Create a bucket for S3 Entry Grants.
Create an IAM coverage and function for S3 Entry Grants.
Arrange S3 Entry Grants.
Enable cross-account entry of sources.
Create Redshift tables.
Unload and cargo knowledge in Amazon Redshift.

Conditions

It’s best to have the next stipulations already arrange:

Allow S3 Entry Grants from the Amazon Redshift managed software

After you’ve created your Redshift software in IAM Id Middle, it’s essential to carry out the next steps to allow S3 Entry Grants within the account the place Amazon Redshift exists. For this put up, we use Member Account 1:

Log in to the AWS Administration Console as admin.
On the Amazon Redshift console, select IAM Id Middle connection within the navigation pane.
Choose the managed Redshift software and select Edit.
Select Amazon S3 entry grants in Trusted id propagation.
Select Save adjustments.

The next screenshot reveals the up to date configuration.

Determine 2: Redshift managed software

Replace the IAM function permission connected to the Amazon Redshift managed software

The Amazon Redshift managed software has an IAM function connected (within the previous screenshot, you’ll be able to see the function referred to as IAMIDCRedshiftRole underneath IAM function for IAM Id Middle entry. We now want to change the coverage on this function and add permissions to permit interplay with Amazon S3. Edit the function and add s3:GetAccessGrantsInstanceForPrefix and s3:GetDataAccess as proven within the following coverage:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "AllowGetRedsfhitInformation",
            "Effect": "Allow",
            "Action": [
                "redshift-serverless:ListNamespaces",
                "redshift-serverless:ListWorkgroups",
                "redshift:DescribeQev2IdcApplications",
                "redshift-serverless:GetWorkgroup"
            ],
            "Useful resource": "*"
        },
        {
            "Sid": "AllowDescribeIdentityCenter",
            "Impact": "Enable",
            "Motion": [
                "sso:DescribeApplication",
                "sso:DescribeInstance"
            ],
            "Useful resource": [
                "arn:aws:sso:::instance/",
                "arn:aws:sso:::application//*"
            ]
        },
        {
            "Sid": "RetrieveAGinstanceforParticularPrefix",
            "Impact": "Enable",
            "Motion": 
                      "s3:GetAccessGrantsInstanceForPrefix",
            "Useful resource": "*"
        },
        {
            "Sid": "CrossAccountAccessGrantsPolicy",
            "Impact": "Enable",
            "Motion": [
                "s3:GetDataAccess"
            ],
            "Useful resource": "arn:aws:s3:::access-grants/default"
        }
    ]
}

Exchange along with your IAM Id Middle occasion ID and with the account ID the place IAM Id Middle is ready up. You additionally want to interchange the useful resource in CrossAccountAccessGrantscasePolicy along with your S3 Entry Grants occasion data.

Create an S3 bucket for S3 Entry Grants

On this step, you create a S3 bucket that you simply need to grant entry to or use an present bucket. For this put up, we create a bucket referred to as amzn-s3-demo-bucket. You may select one other applicable title. For extra data, see Making a common objective bucket.

The bucket have to be situated in the identical AWS Area as your S3 Entry Grants occasion and IAM Id Middle.

Subsequent, create two folders within the newly created S3 bucket. When you’re utilizing an present S3 bucket, establish two folders to make use of for this walkthrough. For this weblog put up, we create two folders: awssso-sales and awssso-finance, underneath a bucket named amzn-s3-demo-bucket. The aim of making two folders is in order that customers from totally different teams have entry solely to their respective folder.

Create an IAM coverage and function for S3 Entry Grants

Full the next steps to create an IAM coverage to scope the permissions for a particular entry grant:

Create an IAM coverage with the next permissions. For extra data on creating IAM coverage, see Create IAM insurance policies. To get further data on the next particular coverage, confer with Register a location.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "ObjectLevelReadPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectAcl",
                "s3:GetObjectVersionAcl",
                "s3:ListMultipartUploadParts"
            ],
            "Useful resource": "arn:aws:s3:::/*",
            "Situation": {
                "StringEquals": {
                    "aws:ResourceAccount": ""
                },
                "ArnEquals": {
                    "s3:AccessGrantsInstanceArn": [
                        "arn:aws:s3:::access-grants/default"
                    ]
                }
            }
        },
        {
            "Sid": "ObjectLevelWritePermissions",
            "Impact": "Enable",
            "Motion": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:AbortMultipartUpload"
            ],
            "Useful resource": "arn:aws:s3:::/*",
            "Situation": {
                "StringEquals": {
                    "aws:ResourceAccount": ""
                },
                "ArnEquals": {
                    "s3:AccessGrantsInstanceArn": "arn:aws:s3:::access-grants/default"
                }
            }
        },
        {
            "Sid": "BucketLevelReadPermissions",
            "Impact": "Enable",
            "Motion": [
                "s3:ListBucket"
            ],
            "Useful resource": "arn:aws:s3:::",
            "Situation": {
                "StringEquals": {
                    "aws:ResourceAccount": ""
                },
                "ArnEquals": {
                    "s3:AccessGrantsInstanceArn": "arn:aws:s3:::access-grants/default"
                }
            }
        }
    ]
}

Create an IAM function that has permission to entry your S3 knowledge within the Area. For extra data, see IAM function creation. On this instance, we create an IAM function referred to as iamidcs3accessgrant. It’s essential to connect the previous coverage to the IAM function.

Use the next belief coverage for the IAM function:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "ForAccessGrants",
            "Effect": "Allow",
            "Principal": {
                "Service": "access-grants.s3.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:SetContext",
                "sts:SetSourceIdentity"
            ],
            "Situation": {
        "StringEquals": {
          "aws:SourceAccount":"",
          "aws:SourceArn":"arn:aws:s3:::access-grants/default"
        }
      }
        }
    ]
}

Arrange S3 Entry Grants

The S3 Entry Grants occasion serves because the container to your S3 Entry Grants sources, which embrace registered places and grants. You may create just one S3 Entry Grants occasion per Area per account. You may affiliate this S3 Entry Grants occasion to your company listing along with your IAM Id Middle occasion. After you’ve completed so, you’ll be able to create grants to your company customers and teams. S3 Entry Grants requires registering a location to map an S3 bucket or prefix to an IAM function, enabling safe entry by offering momentary credentials to grantees for that particular location.

Full the next steps to arrange S3 Entry Grants:

On the Amazon S3 console, select your most popular Area.
Within the navigation pane, select Entry Grants.
Select Create S3 Entry Grants occasion.
Choose Add IAM Id Middle occasion in and enter the IAM Id Middle occasion Amazon Useful resource Identify (ARN). For this put up, we use the delegated administration account IAM Id Middle ARN.
Select Subsequent.

Determine 3: S3 Entry Grants occasion
After you create an Amazon S3 Entry Grants occasion in a Area in your account, you register an Amazon S3 location in that occasion. For Location scope, select Browse S3 or enter the S3 URI path to the situation that you simply need to register. After you enter a URI, you’ll be able to select View to browse the situation. On this instance, we offer the scope as s3://amzn-s3-demo-bucket.
For IAM function, choose Select from present IAM roles and select the IAM function you beforehand created (iamidcs3accessgrant).
Select Subsequent.

This can register a location in your S3 Entry Grants occasion.

Figure 4: S3 Access Grants instance location scope

Determine 4: S3 Entry Grants occasion location scope

You’ll now create a grant.
1. When you chosen the default Amazon S3 location, use the Subprefix field to slim the scope of the entry grant. For extra data, see Working with grants in S3 Entry Grants.
2. When you’re granting entry solely to an object, choose Grant scope is an object. In our instance, we register the situation as s3://amzn-s3-demo-bucket after which for the subprefix, we specify the folder title adopted by an asterisk (awssso-sales/*).
Underneath Permissions and entry, choose the Permission degree, both Learn, Write, or each. On this instance, we choose each as a result of we’ll first unload from Amazon S3 to Amazon Redshift after which copy from the identical bucket to Amazon Redshift.
For Grantee kind, select Listing id from IAM Id Middle.
For Listing id kind, you’ll be able to select both Consumer or Group. On this instance, we select Group.
For IAM Id Middle group ID, enter the group ID from IAM Id Middle the place consumer and group data belongs.

To get this worth, open the IAM Id Middle console and select Teams within the navigation pane, then select one of many teams you need to present entry and duplicate the worth underneath Group ID. Within the following instance, we accumulate the group ID data from the delegated administration account.

Figure 5: IAM Identity Center group information

Determine 5: IAM Id Middle group data

Select Subsequent.

Determine 6: S3 Entry Grants occasion permissions and entry
Select End.

Determine 7: S3 Entry Grants occasion assessment data web page

You may view the main points of the entry grant on the Amazon S3 console, as proven within the following screenshot. For extra data, see View a grant.

Determine 8: S3 Entry Grants grants

Equally, you will get the main points of a location that’s registered in your S3 Entry Grants occasion. For extra data, see View the main points of a registered location.

Determine 9: S3 Entry Grants places

Enable cross-account entry of sources and create preliminary tables

Now we need to share sources to make our cross-account state of affairs work. This step is simply wanted in case your Amazon Redshift and Amazon S3 sources are in several accounts. This must be completed within the account the place Amazon S3 is ready up. Full the next steps:

On the AWS RAM console, within the navigation pane, select Useful resource shares.
Select Create useful resource share.
For Identify, enter a descriptive title for the useful resource share (for instance, s3accessgrant).
For Assets – non-compulsory, select S3 Entry Grants. The S3 Entry Grants occasion you created will probably be proven; choose the default S3 Entry Grant occasion ARN.
Select Subsequent.
Underneath Managed permission for s3:AccessGrants, you’ll be able to select to affiliate a managed permission created by AWS with the useful resource kind, select an present buyer managed permission, or create your personal buyer managed permission for supported useful resource sorts. On this put up, we select the present permission named AWSRAMPermissionAccessGrantsData.
Select Subsequent.
For Grant entry to principals, select Enable sharing solely inside your group and enter the account ID the place the Redshift occasion exists.
Select Add.
Select Subsequent.
Select Create useful resource share.

The next screenshot reveals the brand new useful resource share particulars.

Figure 10: AWS RAM - create resource share wizard

Determine 10: AWS RAM – create useful resource share wizard

Create tables in Amazon Redshift

As an Amazon Redshift admin consumer, it’s essential to first create the tables you’ll use to unload knowledge. Within the following code, we create a brand new store_sales_s3access desk:

CREATE TABLE IF NOT EXISTS 
sales_schema.store_sales_s3access ( 
ID INTEGER ENCODE az64, 
Product varchar(20), 
Sales_Amount INTEGER ENCODE az64 
) 
DISTSTYLE AUTO ;

Additionally ensure the next permissions are utilized on the respective IAM Id Middle group; this group is represented in Amazon Redshift as a Redshift function. For this put up, we grant permissions to the awssso-sales group:

grant utilization on schema sales_schema to function "awsidc:awssso-sales";
grant choose,insert  for tables in schema sales_schema to function "awsidc:awssso-sales";

As an Amazon Redshift admin consumer, you’ve created a Redshift desk and assigned related permissions to the Redshift database function awsidc:awssso-sales. Now when an authenticated consumer that belongs to the group awssso-sales runs a question in Amazon Redshift to entry Amazon S3 (similar to a COPY, UNLOAD, or Amazon Redshift Spectrum operation), Amazon Redshift retrieves momentary Amazon S3 entry credentials scoped to that IAM Id Middle consumer from S3 Entry Grants. Amazon Redshift then makes use of the retrieved momentary credentials to entry the licensed Amazon S3 places for that question.

Unload and cargo knowledge in Amazon Redshift

On this step, we log in to the Amazon Redshift Question Editor utilizing IAM Id Middle authentication and run an UNLOAD command to unload knowledge from the desk created earlier into the S3 bucket. After that, we run the COPY command to repeat data from Amazon S3 into the identical desk in the identical listing we unloaded the information from.

Full the next steps to entry the Amazon Redshift Question Editor with an IAM Id Middle consumer:

On the Amazon Redshift console, open the Amazon Redshift Question Editor.
Select (right-click) your Redshift occasion and select Create connection.
Select IAM Id Middle as your authentication technique.
A pop-up will seem. As a result of your IdP credentials are already cached, it makes use of the identical credentials and connects to the Amazon Redshift Question Editor utilizing IAM Id Middle authentication.

Now you’re able to run the SQL queries in Amazon Redshift.

Unload knowledge

As a federated consumer, you’ll first run an unload command from the desk store_sales within the bucket s3://amzn-s3-demo-bucket/awssso-sales/.

On this put up, we run an UNLOAD command as a federated IAM Id Middle consumer (Ethan), the place we will probably be unloading the information from a Redshift desk. Exchange the S3 bucket title with the one you created.

UNLOAD ('SELECT * FROM "dev"."sales_schema"."store_sales"')
TO 's3://amzn-s3-demo-bucket/awssso-sales/';

The previous command doesn’t embrace an IAM function ARN. This simplified syntax not solely makes your code extra readable, but in addition reduces the potential for configuration errors. The underlying permissions are dealt with mechanically by way of S3 Entry Grants and trusted id propagation, sustaining strong safety whereas simplifying permissions administration.

Load knowledge

Now we reveal a standard knowledge workflow utilizing the identical federated IAM Id Middle consumer (Ethan), the place we will probably be working the COPY command accessing the identical Amazon S3 location the place we beforehand unloaded our knowledge. Use to following command to load knowledge right into a separate desk referred to as store_sales_s3access:

copy dev.sales_schema.store_sales_s3access 
from 's3://amzn-s3-demo-bucket/awssso-sales/' delimiter '|'

If consumer Ethan tries to unload "sales_schema"."store_sales" in sales_schema to a special folder within the S3 bucket (awssso-finance), they get a permission denied error. It is because entry is managed by S3 Entry Grants, and this consumer doesn’t have a grant to the awssso-finance folder. Use the next command to check the entry denied use case:

UNLOAD ('SELECT * FROM "dev"."sales_schema"."store_sales"')
TO 's3://amzn-s3-demo-bucket/awssso-finance/';

Determine 11: QEv2 question outcome error

IAM Id Middle associated operations are mechanically captured and logged in AWS CloudTrail, providing enhanced visibility and complete audit capabilities. To view detailed error data on the CloudTrail console, select Occasion historical past within the navigation pane, then specify s3.amazonaws.com because the occasion supply and open GetDataAccess.

The next screenshot reveals the snippet from the CloudTrail logs exhibiting that consumer entry is denied.

Determine 12: Amazon CloudTrail

Clear up

Full the next steps to scrub up your sources:

Delete the IdP functions that you simply created to combine with IAM Id Middle.
Delete the IAM Id Middle configuration.
Delete the Redshift software and the Amazon Redshift provisioned cluster or serverless occasion that you simply created for testing.
Delete the IAM function and IAM insurance policies that you simply created on this put up.
Delete the permission set from IAM Id Middle that you simply created for the Amazon Redshift Question Editor within the administration account.
Delete the S3 bucket and related S3 Entry Grants occasion.

Conclusion

On this put up, we explored the best way to combine Amazon Redshift with S3 Entry Grants utilizing IAM Id Middle. We established cross-account entry to allow centralized consumer authentication by way of IAM Id Middle within the delegated administrator account, whereas maintaining Amazon Redshift and Amazon S3 remoted by enterprise unit in separate member accounts. We additionally confirmed simplified variations of working COPY and UNLOAD instructions as a federated IAM Id Middle consumer with out utilizing an IAM function ARN. This setup creates a sturdy and safe analytics surroundings that streamlines knowledge entry for enterprise customers.

For extra steerage and detailed documentation, confer with the next key sources:

Concerning the Authors

Maneesh Sharma is a Senior Database Engineer at AWS with greater than a decade of expertise designing and implementing large-scale knowledge warehouse and analytics options. He collaborates with numerous Amazon Redshift Companions and clients to drive higher integration.

Laura is an Id Options Architect at AWS, the place she thrives on serving to clients overcome safety and id challenges. In her free time, she enjoys wreck diving and touring around the globe.

Praveen Kumar Ramakrishnan is a Senior Software program Engineer at AWS. He has almost 20 years of expertise spanning numerous domains together with filesystems, storage virtualization and community safety. At AWS, he focuses on enhancing the Redshift knowledge safety.

Yanzhu Ji is a Product Supervisor within the Amazon Redshift workforce. She has expertise in product imaginative and prescient and technique in industry-leading knowledge merchandise and platforms. She has excellent ability in constructing substantial software program merchandise utilizing net growth, system design, database, and distributed programming strategies. In her private life, Yanzhu likes portray, images, and enjoying tennis.