Deprecation of Lake Formation’s Ruled Tables Characteristic

After cautious consideration, we’ve made the choice to finish help for Ruled Tables, efficient December 31, 2024, to concentrate on open supply transactional desk codecs similar to Apache Iceberg, Apache Hudi, and Linux Basis Delta Lake. This resolution stems from buyer desire for these open supply options, which supply ACID-compliant transactions, compaction, time journey, and different options beforehand offered by Ruled Tables. Amazon Internet Companies (AWS) has enhanced our help for these codecs throughout numerous analytics providers, together with Amazon Athena, Amazon EMR (Elastic MapReduce), AWS Glue, and Amazon Redshift, with options that embody automated compaction help for Apache Iceberg, retention and snapshot expiration and orphan file deletion for Apache Iceberg, enhancements to AWS Glue Information Catalog CreateTable API to create Apache Iceberg tables, and AWS Glue Crawlers schema detection help throughout Apache Iceberg, Apache Hudi, and Linux Basis Delta Lake. Clients can now use these open supply codecs to realize ACID-compliant transactions with Amazon Easy Storage Service (Amazon S3) backed information, benefiting from their wealthy options and vast compatibility.

After December 31, 2024, clients will now not have the ability to create Ruled Tables transactions (lakeformation:StartTransaction), write to Ruled Tables (lakeformation:UpdateTableObjects), or question your Ruled Tables utilizing Amazon Athena. Clients will nonetheless have the ability to entry their desk state info by calling the lakeformation:GetTableObjects and transaction info by calling lakeformation:ListTransactions till February 17, 2025. After February 17, 2025, all Ruled Desk APIs will begin to fail. Ruled Tables metadata will live on inside the AWS Glue Information Catalog, and the Ruled Tables information will stay in your S3 buckets. No different desk varieties can be affected by this variation, together with Hive (Apache Parquet, CSV, ORC, and so forth), Iceberg, Hudi, and Delta Lake tables.

Migrating your Ruled Tables

Clients can migrate their tables from Ruled Tables to one of many open supply codecs by copying their ruled desk information on to Apache Iceberg utilizing Amazon Athena. Emigrate information to Iceberg, you need to use the Amazon Athena CREATE TABLE AS (CTAS) assertion, as proven within the following code instance.

CREATE TABLE my_iceberg_table WITH ( 
    table_type="ICEBERG",
    is_external = false,
    location = 's3://mybucket/myicebergdata/'
) AS 
SELECT * FROM my_governed_table

You may specify further table-level properties, that are listed within the Amazon Athena Person Information. Should you specify partitions or buckets as a part of the Apache Iceberg desk definition, then chances are you’ll run into the 100 partition per bucket limitation. On this case, seek advice from Use CTAS and INSERT INTO to work across the 100 partition restrict.

Should you require any help migrating your tables, or have any questions, attain out to us at governed-tables-support@amazon.com.

Concerning the writer

Mert Hocanin is a Principal Huge Information Architect with AWS Lake Formation.