Asserting the Normal Availability of Row and Column Stage Safety with Databricks Unity Catalog

We’re excited to announce the final availability of Row Filters and Column Masks in Unity Catalog on AWS, Azure, and GCP! Managing fine-grained entry controls on rows and columns in tables is crucial to make sure information safety and meet compliance. With Unity Catalog, you need to use customary SQL capabilities to outline row filters and column masks, permitting fine-grained entry controls on rows and columns. Row Filters allow you to management which subsets of your tables’ rows are seen to hierarchies of teams and customers inside your group. Column Masks allow you to redact your desk values primarily based on the identical dimensions.

“Unity Catalog allowed us to create a unified view of our information property, simplifying collaboration throughout groups inside BlackBerry. We now have a typical method to handle entry permissions and audit information or tables in our lake, with the power to outline fine-grained entry controls on rows and columns. Automated information lineage helped us see the place the info is coming from to pinpoint the supply of a possible menace and to grasp which analysis tasks or groups are leveraging the info for menace detection.”

— Justin Lai, Distinguished Knowledge Architect, Blackberry

This weblog discusses how one can allow fine-grained entry controls utilizing Row Filters and Column Masks.

What’s Coarse-Grained Object-Stage Safety?

Earlier than this announcement, Unity Catalog already supported object-level safety. For instance, you need to use GRANT and REVOKE SQL instructions over securable objects akin to tables and capabilities to regulate which customers and teams are allowed to examine, question, or modify them:

USE CATALOG primary;
CREATE SCHEMA accounts;
CREATE TABLE accounts.purchase_history(
  amount_cents BIGINT,
  area STRING,
  payment_type STRING,
  purchase_date DATE DEFAULT CURRENT_DATE())
USING DELTA;

We will grant learn entry to the accounts_team:

GRANT SELECT ON TABLE accounts.purchase_history TO accounts_team;

Now, the accounts_team has entry to question (however not modify) the purchase_history desk.

Prior Approaches for Sharing Subsets of Knowledge with Totally different Teams

However what if we’ve got separate accounts groups for various areas?

One technique makes use of dynamic views. You may outline a view particularly meant for consumption by particular consumer(s) or group(s):

CREATE VIEW accounts.purchase_history_emea
AS SELECT amount_cents, payment_type, purchase_date
FROM accounts.purchase_history
WHERE IS_ACCOUNT_GROUP_MEMBER('EMEA');

GRANT SELECT ON VIEW accounts.purchase_history_emea
TO accounts_team_emea;

This includes no information copying, however customers nonetheless have to recollect to question the accounts.purchase_history_emea desk if they’re within the EMEA area or the accounts.purchase_history_apac desk if they’re within the APAC area, and so forth.

Dynamic views from an admin perspective have a legitimate worth proposition for sure instances. Nonetheless, for this instance, some constraints apply:

• Restricted to queries; can not insert or replace information inside views
• Should create and keep quite a few views for every area
• Shared SQL logic is cumbersome to reuse throughout totally different regional groups
• Causes litter within the Catalog Explorer

Along with the above, dynamic views don’t present any safety from downstream customers discovering values of rows scanned from referenced tables after which filtered or aggregated out throughout the view. For instance, customers might craft WHERE clauses that throw errors in response to sure column values with the intention of the question optimizer pushing these operations down throughout the view analysis itself.

As a final resort, we might as an alternative create a each day job to repeat subsets of knowledge into totally different tables and set their permissions accordingly:

-- Create a desk for information from the EMEA area and grant
-- learn entry to the corresponding accounts group.
CREATE TABLE accounts.purchase_history_emea(
  amount_cents INT,
  payment_type STRING,
  purchase_date DATE DEFAULT CURRENT_DATE())
USING DELTA;

GRANT SELECT ON TABLE accounts.purchase_history_emea TO accounts_team_emea;

-- Run this each day to replace the customized desk.
-- Use the day before today to verify all the info is out there earlier than
-- copying it.
INSERT INTO accounts.purchase_history_emea
SELECT * EXCEPT (area) FROM accounts.purchase_history
WHERE area = 'EMEA' AND purchase_date = DATE_SUB(CURRENT_DATE(), 1);

Whereas this method successfully addresses question wants, it comes with drawbacks. By duplicating information, we improve storage and compute utilization. Additionally, the duplicated information lags behind the unique, introducing staleness. Furthermore, this resolution caters solely to queries on account of restricted consumer permissions, limiting write entry to the first desk.

Introducing Row Filters

With row filters, you may apply predicates to a desk, guaranteeing that solely rows assembly particular standards are returned in subsequent queries.

Every row filter is applied as a SQL user-defined operate (UDF). To start, write a SQL UDF with a boolean end result whose parameter kind(s) are the identical because the column(s) of your goal desk that you simply wish to management entry by.

For consistency, let’s proceed utilizing the area column of the earlier accounts.purchase_history desk for this goal.

CREATE FUNCTION accounts.purchase_history_row_filter(area STRING)
RETURN CASE
  WHEN IS_ACCOUNT_GROUP_MEMBER('emea') THEN area = 'EMEA'
  WHEN IS_ACCOUNT_GROUP_MEMBER('admin') THEN TRUE
  ELSE FALSE
END;

We will take a look at this logic by performing just a few queries over the goal desk and making use of the operate straight. For somebody within the accounts_team_emea group, such a question would possibly appear like this:

SELECT amount_cents,
  area,
  accounts.purchase_history_row_filter(area) AS filtered 
FROM accounts.purchase_history;

+--------------+--------+----------+
| amount_cents | area | filtered |
+--------------+--------+----------+
| 42           | EMEA   | TRUE     |
| 1042         | EMEA   | TRUE     |
| 2042         | APAC   | FALSE    |
+--------------+--------+----------+

Or for somebody within the admin group who’s setting the entry management logic within the first place, we discover that every one rows from the desk are returned:

SELECT amount_cents, area, purchase_history_row_filter(area) AS filtered 
FROM accounts.purchase_history;

+--------------+--------+----------+
| amount_cents | area | filtered |
+--------------+--------+----------+
| 42           | EMEA   | TRUE     |
| 1042         | EMEA   | TRUE     |
| 2042         | APAC   | TRUE     |
+--------------+--------+----------+

Now we’re prepared to use this logic to our goal desk as a coverage operate, and grant learn entry to the accounts_team_emea group:

ALTER TABLE accounts.purchase_history
SET ROW FILTER accounts.purchase_history_row_filter ON (area);

GRANT SELECT ON TABLE accounts.purchase_history TO accounts_team_emea;

Or, we are able to assign this coverage on to the desk at creation time to verify there is no such thing as a interval the place the desk exists, however the coverage doesn’t but apply:

CREATE TABLE accounts.purchase_history_emea(
  amount_cents INT,
  payment_type STRING,
  purchase_date DATE DEFAULT CURRENT_DATE())
USING DELTA
WITH ROW FILTER purchase_history_row_filter ON (area);

GRANT SELECT ON TABLE accounts.purchase_history TO accounts_team_emea;

After that, querying from the desk ought to return the subsets of rows similar to the outcomes of our testing above. For instance, the accounts_team_emea members will obtain the next end result:

SELECT amount_cents, area FROM accounts.purchase_history;

+--------------+--------+
| amount_cents | area |
+--------------+--------+
| 42           | EMEA   |
| 1042         | EMEA   |
+--------------+--------+

However what if we wish to write new information to the desk? This may not be potential if accounts.purchase_history was a dynamic view, however since it’s a desk with a row filter, we are able to merely use SQL to insert new rows as wanted:

INSERT INTO accounts.purchase_history(amount_cents, area)
VALUES (1043, 'EMEA');

SELECT amount_cents, area FROM accounts.purchase_history;

+--------------+--------+
| amount_cents | area |
+--------------+--------+
| 42           | EMEA   |
| 1042         | EMEA   |
| 1043         | EMEA   |
+--------------+--------+

Now, we are able to share the identical accounts.purchase_history desk with totally different teams with out copying the info or including many new names into our namespace.

You may view this info on the Catalog Explorer. Wanting on the purchase_history desk, we see {that a} row filter applies:

Clicking on the row filter, we are able to see the coverage operate title:

Following the “view” button reveals the operate contents:

Introducing Column Masks

We’ve demonstrated how you can create and apply fine-grained entry controls to tables utilizing row filters, selectively filtering out rows that the invoking consumer doesn’t have entry to learn at question time. However what if we wish to management entry to columns as an alternative, eliding some column values and leaving others intact inside every row?

Asserting column masks!

Every column masks can be applied as a SQL user-defined operate (UDF). Nonetheless, in contrast to row filter capabilities returning boolean outcomes, every column masks coverage operate accepts one argument and returns the identical kind as this enter argument. For instance, we are able to create column masks to filter out PII in e-mail addresses with insurance policies like this:

CREATE FUNCTION email_mask(e-mail STRING)
RETURN CASE
  WHEN IS_ACCOUNT_GROUP_MEMBER('admin') THEN e-mail
  ELSE SUBSTRING(
    SPLIT_PART(e-mail, "@", 1), 1, 1) || "####" || "@" ||
    SPLIT_PART(e-mail, "@", 2)
END;

For our working accounts.purchase_history desk right here, let’s go forward and masks out the acquisition quantity column when the worth is multiple thousand:

CREATE FUNCTION accounts.purchase_history_mask(amount_cents INT)
RETURN CASE
  WHEN IS_ACCOUNT_GROUP_MEMBER('admin') THEN amount_cents
  WHEN amount_cents < 1000 THEN amount_cents
  ELSE NULL
END;

Now, solely directors have permission to take a look at the acquisition quantities of $10 or better.

Let’s go forward and take a look at the coverage operate. Non-admin customers see this:

SELECT amount_cents,
  accounts.purchase_history_mask(amount_cents) AS masked,
  area
FROM accounts.purchase_history;

+--------------+--------+----------+
| amount_cents | masked | area   |
+--------------+--------+----------+
| 42           | 42     | EMEA     |
| 1042         | NULL   | EMEA     |
| 2042         | NULL   | APAC     |
+--------------+--------+----------+

However directors have entry to view all the info:

SELECT amount_cents,
  accounts.purchase_history_mask(amount_cents) AS masked,
  area
FROM accounts.purchase_history;

+--------------+--------+----------+
| amount_cents | masked | area   |
+--------------+--------+----------+
| 42           | 42     | EMEA     |
| 1042         | 1042   | EMEA     |
| 2042         | 2042   | APAC     |
+--------------+--------+----------+

Seems to be nice! Let’s apply the masks to our desk:

ALTER TABLE accounts.purchase_history
ALTER COLUMN amount_cents
SET MASK accounts.purchase_history_mask;

After that, querying from the desk ought to redact particular column values similar to the outcomes of our testing above. For instance, non-administrators will obtain the next end result:

SELECT amount_cents, area FROM accounts.purchase_history;

+--------------+--------+
| amount_cents | area |
+--------------+--------+
| 42           | EMEA   |
| NULL         | EMEA   |
| NULL         | APAC   |
+--------------+--------+

It really works accurately.

We will additionally examine the values of different columns to make our masking choice. For instance, we are able to modify the operate to take a look at the area column as an alternative of the acquisition quantity:

ALTER TABLE accounts.purchase_history ALTER COLUMN amount_cents DROP MASK;

CREATE FUNCTION accounts.purchase_history_region_mask(
  amount_cents INT,
  area STRING)
RETURN CASE
  WHEN IS_ACCOUNT_GROUP_MEMBER('admin') THEN amount_cents
  WHEN area = 'APAC' THEN amount_cents
  ELSE NULL
END;

Now we are able to apply the masks with the USING COLUMNS clause to specify the extra column title(s) to cross into the coverage operate:

ALTER TABLE accounts.purchase_history
ALTER COLUMN amount_cents
SET MASK accounts.purchase_history_mask
USING COLUMNS (area);

Thereafter, querying from the desk ought to redact sure column values in a different way for non-administrators:

SELECT amount_cents, area FROM accounts.purchase_history;

+--------------+--------+
| amount_cents | area |
+--------------+--------+
| NULL         | EMEA   |
| NULL         | EMEA   |
| 2042         | APAC   |
+--------------+--------+

We will take a look at the masks by wanting on the desk column within the Catalog Explorer:

Like earlier than, following the “view” button reveals the operate contents:

Storing Entry Management Lists in Mapping Tables

Row filter and column masks coverage capabilities virtually at all times must check with the present consumer and evaluate it in opposition to a listing of allowed customers or test its group memberships in opposition to an express listing of allowed teams. Itemizing these consumer and group allowlists within the coverage capabilities themselves works nicely for lists of cheap sizes. For bigger lists or instances the place we would favor further assurance that the identities of the customers or teams themselves are hidden from view for customers, we are able to benefit from mapping tables as an alternative.

These mapping tables act like customized gatekeepers, deciding which information rows customers or teams can entry in your unique desk. The fantastic thing about mapping tables lies of their seamless integration with reality tables, making your information safety technique simpler.

This method is a game-changer for numerous customized necessities:

• Tailor-made Consumer Entry: You may impose restrictions primarily based on particular person consumer profiles whereas accommodating particular guidelines for consumer teams. This ensures that every consumer sees solely what they need to.
• Dealing with Complicated Hierarchies: Whether or not it is intricate organizational constructions or various units of guidelines, mapping tables can navigate the complexities, guaranteeing that information entry adheres to your distinctive hierarchy.
• Seamless Exterior Mannequin Replication: Replicating advanced safety fashions from exterior supply techniques turns into a breeze. Mapping tables assist you mirror these intricate setups with out breaking a sweat.

For instance:

CREATE TABLE accounts.purchase_history_groups
AS VALUES ('emea'), ('apac') t(group);

CREATE OR REPLACE FUNCTION accounts.purchase_history_row_filter(area STRING)
RETURN EXISTS(SELECT 1 FROM accounts.purchase_history_groups phg
WHERE IS_ACCOUNT_GROUP_MEMBER(phg.group));

Now, we are able to lengthen the accounts.purchase_history_groups desk to massive numbers of teams with out making the coverage operate itself advanced, and likewise limit entry to the rows of that desk to solely the administrator that created the accounts.purchase_history_row_filter SQL UDF.

Utilizing Row and Column Stage Safety with Lakehouse Federation

With Lakehouse Federation, Unity Catalog solves crucial information administration challenges to simplify how organizations deal with disparate information techniques. It means that you can create a unified view of your total information property, structured and unstructured, enabling safe entry and exploration for all customers no matter information supply. It permits environment friendly querying and information mixture via a single engine, accelerating numerous information evaluation and AI purposes with out requiring information ingestion. Moreover, it supplies a constant permission mannequin for information safety, making use of entry guidelines and guaranteeing compliance throughout totally different platforms.

The fine-grained entry controls introduced right here work seamlessly with Lakehouse Federation tables to assist sharing entry to federated tables inside your organizations with customized row and column-level entry insurance policies for various teams. There isn’t a want to repeat information or create many duplicate or related desk/view names in your catalogs.

For instance, you may create a federated connection to an current MySQL database. Then, browse the Catalog Explorer to examine the international catalog:

Contained in the catalog, we discover a mysql_demo_nyc_pizza_rating desk:

Let’s apply our row filter to that desk:

ALTER TABLE mysql_catalog.qf_mysql_demo_database.mysql_demo_nyc_pizza_rating 
SET ROW FILTER primary.accounts.purchase_history_row_filter ON (title);

Wanting on the desk overview afterward, it displays the change:

Clicking on the row filter reveals the title of the operate, similar to earlier than:

Now, queries over this federated MySQL desk will return totally different subsets of rows relying on every invoking consumer’s id and group memberships. We have efficiently built-in fine-grained entry management with Lakehouse Federation, leading to simplified usability and unified governance for Delta Lake and MySQL tables in the identical group.

Getting began with Row and Column Stage Safety

With Row Filters and Column Masks, you now achieve the ability to streamline your information administration, eliminating extreme ETL pipelines and information copies. That is your gateway to a brand new world of unified information safety, the place you may confidently share information with a number of customers and teams whereas sustaining management and guaranteeing that delicate info stays protected.

To get began with Row Filters and Column Masks, take a look at our documentation on AWS, Azure, and GCP. You may run workloads that question tables with Row Filters and Column Masks on any UC Compute: Serverless, shared entry mode, Single-user entry mode (from DBR 15.4 onwards)