Connect to S3 in PySpark | Dataford Interview Questions

Write the syntax to connect to an S3 bucket in PySpark.

Asked in the Technical interview stage. Implement build_s3_connection to return the s3a:// path and Hadoop configuration needed by a Spark session. Use the S3A filesystem, the supplied credentials, and the supplied AWS region.

Input and Output

The function receives four strings: bucket name, access key, secret key, and region. Return a dictionary containing path and config.

Constraints

bucket, access_key, secret_key, and region are non-empty strings
The returned path must use the s3a:// scheme
The returned configuration must contain all required S3A keys

Function Signature

def build_s3_connection(bucket, access_key, secret_key, region):

Your question is Connect to S3 in PySpark. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem