Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables

Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables

Original Japanese article: Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin). Most of the time, when working with Iceberg tables, we reach for PyIceberg or Spark. I'm no exception, and honestly there were parts of the PyIceberg configuration - rest.sigv4-enabled, rest.signing-name, warehouse - that I understood only vaguely.

Iceberg defines a standard called the Iceberg REST Catalog Open API specification, and AWS implements it through two separate endpoints:

• The AWS Glue Iceberg REST endpoint (https://glue.<region>.amazonaws.com/iceberg)
• The Amazon S3 Tables Iceberg REST endpoint (https://s3tables.<region>.amazonaws.com/iceberg)

If two implementations follow the same spec, sending the same requests to both and comparing the results should reveal what's actually different between them. In this article, I'll bypass clients like PyIceberg entirely and hit the REST API directly to explore the differences between the two endpoints.

To state the conclusion up front: Even though both implement the same Iceberg REST Catalog specification, Glue is designed as an "entry point to multiple catalogs," while S3 Tables is designed as an "entry point to a single table bucket." That difference is visible just by looking at the URL paths.

I previously wrote about the relationship between S3 Tables and Glue Data Catalog in another article - worth a read alongside this one: Does Amazon S3 Tables Replace AWS Glue Data Catalog? Understanding Their Relationship

What Is the Iceberg REST Catalog?

The Iceberg REST Catalog is a specification that standardizes Iceberg catalog operations as an HTTP API. It's published as an OpenAPI definition (YAML), and any catalog that conforms to it can be accessed the same way from clients such as PyIceberg, Spark, and Trino.

The key points of the spec are:

• URL paths follow a pattern like GET /v1/{prefix}/namespaces, where {prefix} is a free-form segment
• Clients first call GET /v1/config to retrieve endpoint configuration (the default prefix and other settings)
• The table metadata body (schema, snapshots, etc.) is returned as JSON in the LoadTable response

In other words, when you write catalog.load_table("ns.table") in PyIceberg, what's actually happening under the hood is an HTTP request to GET /v1/{prefix}/namespaces/ns/tables/table.

Here's a summary of the two AWS implementations before we dive in:

Here's the overall picture as a diagram:

Iceberg REST Catalog spec
GET /v1/{prefix}/namespaces/...
│
┌───────────────┴───────────────┐
│                               │
Glue endpoint          S3 Tables endpoint
│                               │
prefix = /catalogs/{catalog}    prefix = table bucket ARN
(multi-catalog hierarchy)       (one bucket = one catalog)
│                               │
IAM + Lake Formation            s3tables IAM actions
│                               │
└───────────────┬───────────────┘
                │
    Same Iceberg table
    (points to the same metadata.json)

Note: a catalog doesn't manage the metadata.json file itself - it provides a reference to where the latest metadata-location is. The catalog's essential job is knowing where the current metadata.json lives.

Setting Up the Test Environment

Let's create a test table bucket, namespace, and table via the CLI. Read the region as Tokyo (ap-northeast-1) and the account ID as 123456789012.

# Create a table bucket
aws s3tables create-table-bucket \
    --name penguin-rest-test \
    --region ap-northeast-1

# Create a namespace
aws s3tables create-namespace \
    --table-bucket-arn arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test \
    --namespace analytics

# Create a table (with schema)
aws s3tables create-table \
    --table-bucket-arn arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test \
    --namespace analytics \
    --name daily_sales \
    --format ICEBERG \
    --metadata '{
        "iceberg": {
            "schema": {
                "fields": [
                    {"name": "sales_date", "type": "date", "required": false},
                    {"name": "amount", "type": "long", "required": false}
                ]
            }
        }
    }'

SigV4 Signing

You can't hit these endpoints with plain curl. The Iceberg REST Catalog spec defines an OAuth2-based authentication flow, but AWS's implementation uses IAM SigV4 signing instead - a standard-spec API with AWS-flavored authentication. That rest.sigv4-enabled: true setting in PyIceberg is exactly what enables this signing.

Computing SigV4 signatures by hand is painful, so I used awscurl for this exercise.

pip install awscurl

awscurl picks up credentials from environment variables or a profile and sends SigV4-signed requests for you. The important part is the --service option, which must specify the correct service name for signing: glue for the Glue endpoint, s3tables for the S3 Tables endpoint.

Hitting the S3 Tables Iceberg REST Endpoint

Let's start with the simpler of the two - the S3 Tables endpoint.

GET /v1/config

getConfig is the first API a REST Catalog client calls. The warehouse query parameter takes the table bucket ARN.

Note: When using awscurl, pass the raw, unencoded ARN as the query parameter value. awscurl automatically URL-encodes query parameters before computing the signature, so if you pre-encode the value yourself, it gets double-encoded and you'll get a SignatureDoesNotMatch error.

BUCKET_ARN="arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test"

awscurl --service s3tables --region ap-northeast-1 \
    "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/config?warehouse=${BUCKET_ARN}"

Result (excerpt):

{
    "defaults": {
        "prefix": "arn%3Aaws%3As3tables%3Aap-northeast-1%3A123456789012%3Abucket%2Fpenguin-rest-test",
        "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
        "write.object-storage.enabled": "true",
        "write.object-storage.partitioned-paths": "false",
        "s3.delete-enabled": "false",
        "rest-metrics-reporting-enabled": "false"
    },
    "overrides": {}
}

The interesting part is the prefix in the response. For the S3 Tables endpoint, prefix is exactly the URL-encoded table bucket ARN, and it lives under defaults. In other words, this endpoint is designed around a "one table bucket = one catalog" model - if you want to work with multiple table buckets, you register multiple catalogs on the client side.

defaults also included the write-side FileIO implementation (io-impl), object storage layout settings (write.object-storage.*), and a setting that disables S3 delete operations (s3.delete-enabled). It's interesting to see, from actual output, that some of the parameters we're used to configuring individually on the client (PyIceberg) side are actually being pushed down as server-side defaults through this config response.

Incidentally, this getConfig call is authorized under the s3tables:GetTableBucket IAM action. The official documentation includes a mapping table showing which s3tables IAM action corresponds to each REST operation.

Note: The spec's CatalogConfig also defines an endpoints field, which lets the server return a list of supported endpoints (in a format like "GET /v1/{prefix}/namespaces"). When I checked, the S3 Tables endpoint's response did not include an endpoints field.

Listing Namespaces and Tables

Now that we know the prefix, let's list namespaces and tables.

Note: There's a subtlety worth calling out here. The warehouse query parameter for GET /v1/config needs the raw ARN, since awscurl automatically URL-encodes query parameters before signing - pre-encoding it yourself causes double-encoding and a SignatureDoesNotMatch error. The {prefix} segment in the path, however, behaves differently. If you put the ARN into the path completely unencoded (colons and slashes as-is), the / characters inside the ARN get interpreted as path separators, and the request no longer matches the modeled URL pattern /v1/{prefix}/namespaces - you get an UnknownOperationException. The correct approach is to leave the colons raw but percent-encode only the single / inside bucket/<table-bucket-name> as %2F. That makes the request match the correct route (and once it reaches the authorization layer, you'll get a 403 if you lack permissions, rather than a routing error). Even though it's the same operation - putting an ARN into a URL - the encoding rules differ between query parameters and path segments. That's something you only really notice by actually hitting the endpoint.

BUCKET_ARN_PATH="arn:aws:s3tables:ap-northeast-1:123456789012:bucket%2Fpenguin-rest-test"

# List namespaces
awscurl --service s3tables --region ap-northeast-1 \
    "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces"

# List tables
awscurl --service s3tables --region ap-northeast-1 \
    "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables"

Result (excerpt):

{
    "namespaces": [
        ["analytics"]
    ]
}

{
    "identifiers": [
        {
            "name": "daily_sales",
            "namespace": ["analytics"]
        }
    ]
}

There's one difference from the spec worth noting here: the Iceberg REST spec allows multi-level namespaces (like a.b.c), but S3 Tables only supports a single level. Let's confirm this by trying to create a multi-level namespace:

awscurl --service s3tables --region ap-northeast-1 \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{"namespace": ["level1", "level2"]}' \
    "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces"

Result (excerpt):

{
    "error": {
        "code": 400,
        "message": "Multipart namespaces are not supported.",
        "type": "bad_request"
    }
}

The error message spells it out directly - "multipart namespaces are not supported" - confirming the single-level restriction empirically.

Retrieving Metadata with LoadTable

This is the API I most wanted to check today.

awscurl --service s3tables --region ap-northeast-1 \
    "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables/daily_sales"

Result (excerpt):

{
    "metadata-location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json",
    "metadata": {
        "format-version": 2,
        "table-uuid": "9507ea2b-7a71-40a7-b6a9-1d6e633955e1",
        "location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3",
        "current-schema-id": 0,
        "schemas": [
            {
                "type": "struct",
                "schema-id": 0,
                "fields": [
                    { "id": 1, "name": "sales_date", "required": false, "type": "date" },
                    { "id": 2, "name": "amount", "required": false, "type": "long" }
                ]
            }
        ],
        "default-spec-id": 0,
        "partition-specs": [{ "spec-id": 0, "fields": [] }],
        "properties": { "write.parquet.compression-codec": "zstd" },
        "current-snapshot-id": -1,
        "snapshots": []
    },
    "config": {
        "tableBucketId": "3b2a6702-dd99-44f2-bc03-2f9f6b273104",
        "namespaceId": "20576faa-861e-499f-a30f-4d6c7550dd55",
        "tableId": "34c72c19-610d-4e5c-bee4-6ae9f093c583"
    }
}

This JSON is exactly what's behind the information we normally see through table.schema() or table.snapshots() in PyIceberg. The response directly confirms that a catalog's essential job is "managing and returning the S3 path of the latest metadata.json" (metadata-location).

Incidentally, the config field in the response contained S3 Tables-internal identifiers: tableBucketId, namespaceId, and tableId. Per spec, the Glue endpoint's config is supposed to include temporary credentials (vended credentials), but here on the S3 Tables endpoint there's no credential delegation at all - it just returns internal management IDs.

Note: The S3 Tables endpoint also documents a limit: operations against a table whose metadata.json exceeds 50MB return a 400 error. Worth keeping in mind - if metadata bloats from accumulated snapshots and similar growth, you could hit this API-level ceiling.

Checking What's Happening Under the Hood via CloudTrail

An interesting characteristic of the S3 Tables endpoint is that REST API calls get logged in CloudTrail as their corresponding native S3 Tables actions. Per the official documentation, a single LoadTable call logs both of the following:

• GetTableMetadataLocation (a management event)
• A data event corresponding to GetTableData

In other words, the audit log itself reveals that this endpoint is implemented as a proxy that translates the Iceberg REST API into native S3 Tables API calls.

Let's check CloudTrail directly:

aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=GetTableMetadataLocation \
    --region ap-northeast-1 \
    | jq '.Events[0] | {time: .EventTime, user: .Username, event: .EventName}'

Result (excerpt):

{
    "time": "2026-07-09T13:46:44+00:00",
    "user": "penguin-test-user",
    "event": "GetTableMetadataLocation"
}

Even though we only called LoadTable over REST, CloudTrail logs it under the native S3 Tables event name s3tables:GetTableMetadataLocation, complete with the exact caller (IAM username).

Hitting the Glue Iceberg REST Endpoint

Now let's look at the Glue endpoint, which behaves quite differently.

GET /v1/config

awscurl --service glue --region ap-northeast-1 \
    "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/config?warehouse=123456789012"

Result (excerpt):

{
    "defaults": {
        "prefix": "123456789012",
        "header.Content-Type": "application/x-amz-json-1.1",
        "rest.sigv4-enabled": "true",
        "rest.signing-name": "glue",
        "rest.signing-region": "ap-northeast-1",
        "rest-table-scan-enabled": "true",
        "rest-data-commit-enabled": "true",
        "token-refresh-enabled": "false"
    },
    "overrides": {
        "prefix": "catalogs/123456789012"
    }
}

There's a difference from the S3 Tables endpoint worth calling out here: defaults.prefix is "123456789012" (just the account ID), while overrides.prefix