# Building Reusable Checks in a Fabric Notebook

In the [last post](https://buildingagain.hashnode.dev/fabric-s-silver-layer-making-your-data-reliable), we looked at why the Silver layer matters.

A pipeline can run without errors and still contain duplicate records, missing values, or invalid data.

This post will show you how to get started in a simple way to build reusable validation checks in a Microsoft Fabric notebook.

Let's take a look at a small orders table:

1.  We'll run three checks,
    
2.  save the results,
    
3.  and show how a Fabric pipeline can run the notebook.
    

## Reusable Checks

If possible, we want to avoid writing a new check for every single data table we're ingesting. Instead, create a small set of functions that take a `DataFrame` and return a result.

Each check should answer three questions:

*   What did we check?
    
*   How many rows failed?
    
*   What should happen next?
    

That makes the checks easier to reuse and easier to understand later.

```plaintext
Read Silver data
       │
       ▼
Run reusable checks
       │
       ▼
Save validation results
       │
       ▼
Pipeline decides next step
```

## The sample data

Suppose the Bronze layer contains orders from a source system.

This data is loaded into a Fabric Lakehouse as a Delta table called `bronze_orders`.

A Fabric Lakehouse stores tables in **Delta Lake** format by ***default***.

The table has these columns:

| Column | Description |
| --- | --- |
| `order_id` | The order number |
| `customer_id` | The customer who placed the order |
| `order_date` | The date the order was placed |
| `status` | The order status |

The Silver table should contain **valid orders only**.

## Create the notebook

In Fabric, open the Lakehouse and create a new notebook. Attach the Lakehouse that contains the tables.

> **Note:** A Fabric notebook is a code-based workspace where you can use PySpark to read, transform, and write Lakehouse data.

## Create a parameter cell

This lets a pipeline pass values into the notebook later.

In Fabric, select the cell menu and choose Toggle parameter cell. A pipeline’s Notebook activity can pass values through that cell.

![](https://cdn.hashnode.com/uploads/covers/609ec78747373d4cb056441b/3a2212c1-b0a4-4fb2-b5f3-49fc690bbfef.png align="center")

You'll see this updated on the bottom right of the parameter cell:

![](https://cdn.hashnode.com/uploads/covers/609ec78747373d4cb056441b/70a40110-2b92-4054-91c1-d7946d96d643.png align="center")

After you choose Toggle parameter cell:

![](https://cdn.hashnode.com/uploads/covers/609ec78747373d4cb056441b/7b3f90cd-922d-4efa-8129-01fe30f319f2.png align="center")

Enter the parameters in the cell:

```python
# Parameter cell

table_name = "bronze_orders"
silver_table_name = "silver_orders"
validation_table_name = "silver_order_validation_results"
```

> **Note:** When you run the notebook manually, these defaults are used. When a pipeline runs it, the pipeline can replace them.

## Read the Bronze table

Add this cell to read the orders table.

```python
df = spark.read.table(table_name)

display(df)
```

For our example, let's assume that this is what the table looks like:

| order\_id | customer\_id | order\_date | status |
| --- | --- | --- | --- |
| 1001 | 501 | 2026-10-01 | Shipped |
| 1002 | 502 | 2026-10-01 | Pending |
| 1003 | None | 2026-10-02 | Shipped |
| 1004 | 504 | 2026-10-02 | Canceled |
| 1004 | 504 | 2026-10-02 | Canceled |
| 1005 | 505 | 2026-10-03 | Unknown |

The data has three problems:

1.  One order has no customer ID.
    
2.  One order appears twice.
    
3.  One order has a status that is not allowed.
    

## Build the reusable checks

Add this cell. It creates a reusable function that records the result of each check.

```python
from pyspark.sql import functions as F


def run_check(df, check_name, failed_df):
    """
    Run one validation check and return a one-row result.
    """
    total_rows = df.count()
    failed_rows = failed_df.count()

    result = spark.createDataFrame(
        [
            (
                check_name,
                total_rows,
                failed_rows,
                "FAIL" if failed_rows > 0 else "PASS"
            )
        ],
        schema=["check_name", "total_rows", "failed_rows", "result"]
    )

    return result
```

This is a common approach in **Fabric Medallion architectures** during the Silver ("Clean and Validated") layer, where each rule produces a standardized PASS/FAIL result before data moves downstream into Gold datasets.

Keep in mind that this function doesn't decide what happens next. It only records what happened.

That separation matters. You can reuse the same function for different tables and different checks.

## Check required values

Some fields should not be blank. For orders, `order_id`, `customer_id`, and `order_date` are required and must have a value.

```python
# Define a list of columns that are required to contain values.
# This variable serves as documentation and could be reused later
# for dynamic validation logic.

required_columns = ["order_id", "customer_id", "order_date"]

# Create a DataFrame containing only rows that fail the validation.
# Any row where order_id, customer_id, OR order_date is NULL
# will be included in this DataFrame.
missing_values_df = df.filter(
    
    # Check if order_id is NULL
    F.col("order_id").isNull()   
    | 
    # Check if customer_id is NULL
    F.col("customer_id").isNull()
    |    
    # Check if order_date is NULL
    F.col("order_date").isNull()
)

# Run the standardized validation function.
#
# Parameters:
#   df = the original DataFrame being validated
#   "required_values_present" = name of the validation rule
#   missing_values_df   = rows that failed the validation
#
# The function will count:
#   - total rows in df
#   - failed rows in missing_values_df
# and return a one-row PASS/FAIL summary DataFrame.
missing_values_result = run_check(
    df,
    "required_values_present",
    missing_values_df
)

# Display the validation results in the Fabric notebook output.
#
# Example output:
#
# +-------------------------+-----------+------------+--------+
# | check_name              |total_rows |failed_rows |result  |
# +-------------------------+-----------+------------+--------+
# |required_values_present  |6          |1           |FAIL    |
# +-------------------------+-----------+------------+--------+
display(missing_values_result)
```

Before adding a check like this, here are good questions to ask:

*   Which fields are truly required?
    
*   Does a blank mean missing data or a valid business condition?
    
*   Should a missing value stop the batch or be handled separately?
    

For example, a missing `customer_id` might be acceptable for a guest order. This fully depends on the business rules.

## Check for duplicates

Next, check whether the same order appears more than once. Notice how we're using `run_check()` again with this check.

```python
# Find duplicate order IDs in the dataset         

# Group records by order_id   
# Count how many times each order_id appears
# Keep only order_ids that appear more than once

duplicate_keys_df = (
    df
    .groupBy("order_id")
    .count()
    .filter(F.col("count") > 1)
)

# Run the validation check against the duplicate records
duplicate_keys_result = run_check(
    df,                     # Original dataset
    "duplicate_order_ids", # Validation rule name
    duplicate_keys_df       # Failed records
)

# Display the validation result
display(duplicate_keys_result)
```

This check finds duplicate order\_id values.

Before we proceed to delete any duplicates, here are some things to consider:

*   What does one row represent?
    
*   Can one order have more than one line item?
    
*   Which fields make a record unique?
    

In this example, `order_id` is the key. If your table had order lines, you would likely need both `order_id` and `line_number`.

## Check allowed values

Some columns should contain only approved values. Here, we check the order status. Again, we reuse `run_check()` with this additional check.

```python
# Define the list of valid status values
allowed_statuses = ["Pending", "Shipped", "Canceled"]

# Find rows where the status is not in the allowed list
invalid_status_df = df.filter(
    ~F.col("status").isin(allowed_statuses)
)

# Run the validation check
invalid_status_result = run_check(
    df,                        # Original dataset
    "status_in_allowed_list",  # Validation rule name
    invalid_status_df          # Rows that failed the check
)

# Display the validation result
display(invalid_status_result)
```

This check catches the `Unknown` status.

Here are some good things to ask:

*   Who owns the list of valid values?
    
*   Can new statuses appear without notice?
    
*   Should an unexpected value fail the check or be sent to a review table?
    

## Save the results

Now combine the results and save them to the Lakehouse.

```python
# Combine all validation results into a single DataFrame
validation_results = (
    missing_values_result      # Required values check
    .unionByName(duplicate_keys_result)   # Duplicate order ID check
    .unionByName(invalid_status_result)   # Allowed status check
)

# Save the validation results to a Delta table
validation_results.write \
    .mode("overwrite") \       # Replace existing data if the table already exists
    .format("delta") \         # Save in Delta format
    .saveAsTable(validation_table_name)   # Save using the specified table name

# Display all validation results
display(validation_results)
```

The results table looks like this:

| check\_name | total\_rows | failed\_rows | result |
| --- | --- | --- | --- |
| required\_values\_present | 6 | 1 | FAIL |
| duplicate\_order\_ids | 6 | 1 | FAIL |
| status\_in\_allowed\_list | 6 | 1 | FAIL |

This table becomes your quality record. It tells you what was checked, how many records failed, and whether the check passed.

## Create the Silver table

For this simple example, we are separating the failed records instead of writing them to Silver.

```python
# Keep only valid records
valid_orders_df = (
    df    
    # Remove rows where customer_id is missing
    .filter(F.col("customer_id").isNotNull())    
    # Keep only rows with valid status values
    .filter(F.col("status").isin(allowed_statuses))    
    # Remove duplicate order_ids
    .dropDuplicates(["order_id"])
)

# Save the cleaned data to the Silver Delta table
valid_orders_df.write \
    .mode("overwrite") \       # Replace existing table data
    .format("delta") \         # Save in Delta format
    .saveAsTable(silver_table_name)   # Save to the Silver table

# Display the cleaned dataset
display(valid_orders_df)
```

This code block creates the clean Silver dataset by:

1.  Removing records with missing customer\_id values.
    
2.  Keeping only records with valid statuses (Pending, Shipped, or Canceled).
    
3.  Removing duplicate order\_id values.
    
4.  Saving the cleaned data to the Silver Delta table.
    

This is a typical Bronze --> Silver transformation step in a **Fabric Medallion Architecture**.

In a real pipeline, you would also save the **failed records** with a reason for the failure. That makes it easier to investigate and correct them later.

## Run the notebook from a pipeline

Fabric pipelines can run notebooks as part of a larger process. The Notebook activity lets you select a notebook and pass parameters to it.

To set this up:

1.  Open a Data pipeline in Fabric.
    
2.  Add a **Notebook** activity.
    
3.  Select your validation notebook.
    
4.  Open **Settings**.
    
5.  Under **Base parameters**, add values for `table_name`, `silver_table_name`, and `validation_table_name`.  
    
    ![](https://cdn.hashnode.com/uploads/covers/609ec78747373d4cb056441b/e0b1c729-33c4-4d0a-a0a4-67fef1f4de0e.png align="center")
    
6.  Save and run the pipeline.
    

Fabric finds the notebook’s parameter cell and uses it to pass in the values.

Now the same notebook can be reused to validate different tables without the need to change the code.

## Before you add more checks

The best way to start is with a small number of checks. Add more checks based on the data problems you actually see.

Ask yourself:

*   Which problems should stop the entire pipeline?
    
*   Which problems should affect only the records with errors?
    
*   Where should validation results live?
    
*   Who should see a failed check?
    
*   Can a failed record be corrected and processed again?
    
*   Can you reuse the same checks for different tables, or do they work only for one table?
    

You don't need many checks to start. Begin with a few that catch common data problems. Make sure the results are easy to understand and that you can run the checks whenever you need to.