Fabric's Silver Layer: Making Your Data Reliable
Clean Data Is Not Enough
A successful pipeline does not always mean good data.
A pipeline can run without errors and still deliver data you should not trust. That is why the Silver layer deserves more attention than simply “cleaning things up.”
It might load duplicate orders, missing customer IDs, or invalid dates without failing. That is why we need to validate the data, not just check whether the pipeline ran.
Where the Silver layer fits
Medallion architecture organizes data into three layers, progressively improving its structure and quality:
Bronze: Keep the raw data.
Silver: Clean and validate it.
Gold: Prepare it for reporting and analysis.
Silver is where you build trust in the data before other people use it.
Validation and the questions worth asking
Start with these six checks.
1. Check the structure
Make sure the expected columns exist and contain the right types of data.
Are any required columns missing?
Did a number or date arrive as text?
What happens if a value cannot be converted?
2. Check for missing values
Decide which fields must have a value. Not every blank is an error, but every required field needs a clear rule.
Which fields are required?
Is this blank expected?
Should a record with a missing value continue?
For example, a missing customer ID might be acceptable for a guest order, but not for an account payment.
3. Check for duplicates
Define what makes a record unique before removing repeated rows. Deduplication is a common Silver-layer task.
What does one row represent?
Which fields identify a unique record?
If two records conflict, which one should we keep?
An order can have several items. Repeated order IDs do not necessarily mean duplicate data.
4. Check the business rules
A value can have the right data type and still be wrong. Validate whether the data makes sense for the business.
Is this status allowed?
Can a shipment date come before the order date?
Is a negative amount an error or a refund?
Make sure the business rules are defined by subject matter experts (SMEs) who understand the business process.
5. Check that the expected data arrived
Valid rows do not prove that you received all the data. Check arrival times and unexpected changes in row counts.
Did today’s data arrive?
Are there far fewer rows than usual?
Is the change expected, or should we investigate?
6. Decide what happens when a check fails
A validation check needs a response. You might stop the pipeline, raise a warning, or separate the failed records for review.
Should this stop the whole batch?
Can we separate the affected records?
Who will investigate?
Can we process the records again after they are corrected?
What Happens During Validation?
Do not assume that a batch can be published if only some records are valid. Define clear rules for when a failure should stop the entire batch before the pipeline runs.
Do not silently remove invalid records. Always explain which records were excluded and why, so users understand why the totals have changed.
Here's how you can get started
For your first Fabric implementation, pick one dataset and a few checks that matter. A Fabric notebook can run reusable validation checks, with a pipeline coordinating execution and recording the results.
For an orders table, that might mean checking required IDs, valid dates, duplicates, and allowed statuses. Record the results so you can see what passed, what failed, and why. Implement checks that protect the dataset’s intended use and have a clear plan when they fail.
The Silver layer is not just about making data look cleaner. A green pipeline tells you the code ran. Silver-layer validation helps you decide whether the result deserves to move forward.
