We’re excited to announce the Public Preview of Hive Metastore (HMS) and AWS Glue Federation in Unity Catalog! This new functionality allows Unity Catalog to seamlessly entry and govern tables saved in Hive Metastores—whether or not inner to Databricks or exterior—in addition to AWS Glue. It represents a key milestone in our Lakehouse Federation imaginative and prescient, which brings exterior information sources together with databases, information warehouses and catalogs, collectively underneath a unified governance framework with Unity Catalog. You possibly can effortlessly uncover, govern and question all of your information from a single, centralized platform, whatever the format and site. This not solely fosters open entry and collaboration throughout your group but in addition extends information intelligence into each information supply.
On this weblog, we’ll discover the advantages of HMS and AWS Glue Federation, clarify the way it works, and supply steerage on getting began.
Why Hive Metastore and AWS Glue Federation?
HMS has been an early normal for cataloging information to be used in massive information methods, and whereas it supplies foundational functionalities, they aren’t ideally fitted to trendy information and AI workloads that demand complete governance together with fine-grained entry controls on rows and columns, lineage, monitoring and auditing throughout all information and AI property in a single place.
Unity Catalog addresses these shortcomings by offering the business’s solely unified, open governance resolution for managing all information and AI property. It allows organizations to create an enterprise catalog that curates information, tables, ML fashions, AI instruments, notebooks, and metrics, all ruled with fine-grained entry controls, lineage, monitoring, auditing and cross-platform sharing in a single resolution. Over 10,000+ enterprises are actually leveraging Unity Catalog to manipulate their information property.
HMS and AWS Glue Federation present important advantages for organizations with HMS deeply embedded of their information structure. For these with long-standing HMS or AWS Glue deployments, this functionality presents a seamless path to leverage Unity Catalog’s superior options over information saved within the HMS or Glue metastore. It ensures operational continuity by enabling organizations to maintain legacy workflows whereas regularly upgrading present information and workspaces to Unity Catalog.
Key advantages embody:
- Seamless integration: Join your present HMS and AWS Glue catalogs on to Unity Catalog with out requiring guide metadata migration.
- Simplified information discovery: Entry and discover metadata from HMS and AWS Glue by means of a unified interface, alongside different information and AI property in Unity Catalog.
- Complete governance: Leverage Unity Catalog’s fine-grained entry controls, tagging, classification, lineage, and audit capabilities on high of the information saved in HMS and AWS Glue.
“We have now years’ value of datasets which can be cataloged in an exterior Hive Metastore. HMS Federation permits us to right away profit from Unity Catalog solely options like sturdy entry management and self-serve AI tooling by means of Genie Areas, with out the overhead of migrating all of those tables into Unity Catalog”
— James Davidheiser, Technical Lead, Information Infrastructure, Asana
The way it works
Unity Catalog now contains federation connectors for Hive Metastore (HMS) and AWS Glue, serving as a translation layer between Unity Catalog and your exterior metastores. These connectors allow you to mount complete HMS catalogs (each inner and exterior) or AWS Glue as international catalogs inside Unity Catalog, making them seem as native objects. You possibly can outline fine-grained entry controls, view lineage, carry out audits, and question HMS or AWS Glue managed tables utilizing the Databricks engine. The federation helps each studying and writing to tables in inner HMS inside Databricks workspaces whereas providing read-only entry for tables in exterior HMS and AWS Glue.
With this functionality, you may learn all tables in HMS and AWS Glue—Parquet, Delta, and Iceberg (coming quickly in Public Preview)—enabling you to entry and govern all of your tables seamlessly.
Take a look at the video tutorial under to discover AWS Glue and HMS Federation in motion.
Get began
By embracing Unity Catalog because the cornerstone of your Lakehouse structure, you may unlock the facility of a unified and open governance implementation that spans your complete information and AI property.