Glue CreateConnection for BIGQUERY fails: server expects SparkProperties but SDK requires ConnectionProperties

[shubhamdixit863]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

We’re creating an AWS Glue connection with ConnectionType=BIGQUERY via AWS SDK v2. The request must include ConnectionInput.ConnectionProperties per SDK validation, but when sent with ConnectionProperties and SparkProperties, Glue returns:

InvalidInputException: SparkProperties: is missing but it is required

We are sending:

ConnectionType: BIGQUERY
ConnectionProperties: {}
SparkProperties: { "secretId": "<secretName>" }

If we omit ConnectionProperties, the SDK rejects the request with:

missing required field, CreateConnectionInput.ConnectionInput.ConnectionProperties

It appears the service schema for BIGQUERY requires SparkProperties, but the SDK is either not serializing them correctly or the backend expects a different structure for BIGQUERY connections (possibly different property keys).

Exact go code

// Create Glue connection input
	connectionInput := &gluetypes.ConnectionInput{
		Name:           aws.String(connectionName),
		ConnectionType: gluetypes.ConnectionTypeBigquery,
		// Native connectors use SparkProperties for runtime configuration
		SparkProperties: map[string]string{
			"secretId": secretName, // The ARN or name of the secret in Secrets Manager
		},
		// ConnectionProperties often mirror SparkProperties for catalog validation
		ConnectionProperties: map[string]string{
			"SECRET_ID":     secretName,
			"parentProject": "cresto",
		},
		PhysicalConnectionRequirements: &gluetypes.PhysicalConnectionRequirements{
			SubnetId:            aws.String(subnetIdList[0]),
			SecurityGroupIdList: securityGroupIdList,
		},
		Description: aws.String(fmt.Sprintf("Glue connection for BigQuery-catalog %s", catalogName)),
	}

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Should be able to create glue connection with google bigquery

Please clarify the correct payload schema for BIGQUERY connections, and/or update the SDK validation to align with the service.

Current Behavior

not able to create glue connection with google big query

Reproduction Steps

// Create Glue connection input
	connectionInput := &gluetypes.ConnectionInput{
		Name:           aws.String(connectionName),
		ConnectionType: gluetypes.ConnectionTypeBigquery,
		// Native connectors use SparkProperties for runtime configuration
		SparkProperties: map[string]string{
			"secretId": secretName, // The ARN or name of the secret in Secrets Manager
		},
		// ConnectionProperties often mirror SparkProperties for catalog validation
		ConnectionProperties: map[string]string{
			"SECRET_ID":     secretName,
			"parentProject": "crestolympiads-b63c3",
		},
		PhysicalConnectionRequirements: &gluetypes.PhysicalConnectionRequirements{
			SubnetId:            aws.String(subnetIdList[0]),
			SecurityGroupIdList: securityGroupIdList,
		},
		Description: aws.String(fmt.Sprintf("Glue connection for BigQuery-catalog %s", catalogName)),
	}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.41.0

Compiler and Version used

go version go1.25.2 darwin/arm64

Operating System and version

Mac os tahoe 26.1

[wty-Bryant]

Hi, can you provide the full repro code with request/response logging when you call glue.CreateConnection so we can further debug how go v2 glue client serializes those 2 properties based on your input. The instruction for logging is here: https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-logging.html

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.