Skip to main content

Incremental Validation

note

If you haven't read the tutorial on Validators, read it first, then come back to this tutorial.

What you'll achieve​

By the end of this tutorial, you'll understand incremental validation and will run incremental SQL and Content Validation against your Looker instance.

What is incremental validation?​

Incremental validation means that Spectacles only shows you errors that are unique to your branch and don't already exist in production. Running incrementally means that a) you'll often see fewer errors and b) for some validators, you'll get results faster.

info

For example, let's say we're writing some additional LookML on a development branch called feature/period-over-period. If our Looker instance already has a lot of Content Errors on the production branch, running the Content Validator on our development branch probably won't be very helpful.

Why not? It will be difficult to tell which of the many Content Errors are a direct result of our changes and which Content Errors are unrelated, already existing in production.

Running the Content Validator in incremental mode fixes this: Spectacles will only show us the Content Errors that are unique to our branch, hiding all of the errors that already exist in production.

You can enable incremental validation for the SQL and Content validators, but not the LookML or Assert validators. Let's try running some incremental validations.

Validate content incrementally​

In Looker, create a new branch off production called spectacles-tutorial and make a change that introduces a Content Error. The easiest way to do this is to rename a dimension you know some content depends on. Commit the change and push it to remote.

Next, in Spectacles, start an incremental Content Validation against your tutorial branch.

  1. In the top menu, click Suites

  2. Click New Suite

  3. Name your Suite "Incremental Validation"

  4. Disable all validators except the Content Validator

  5. Configure the Content Validator section with the following options:

    • Only incremental errors: Check the box

  6. Click Create Suite

  7. On the Suites page, next to the Suite you just created, click Create Run

    • Under Workspace, choose Dev Mode
    • Under Branch, choose spectacles-tutorial
    • Click Start

Spectacles will start a new run using incremental Content Validation.

Spectacles should identify the Content Errors associated with the LookML change you made, but shouldn't show you any additional, unrelated errors, even if your Production branch has errors.

Change the target branch​

By default, Spectacles compares your desired branch or commit to production. What if you wanted to compare to another branch instead of to production?

In Looker, branch off of spectacles-tutorial and create a new branch called spectacles-tutorial-more. Make a different change to introduce a new Content Error.

When comparing our -more branch to our first branch incrementally, we would only expect to see one error identified by Spectacles: the newest error that we just created.

Let's try it. In Spectacles, we'll start an incremental Content Validation against the spectacles-tutorial branch instead of production.

  1. In the top menu, click Suites

  2. On the Suites page, next to the Suite you created in the previous step, click Create Run

    • Under Workspace, choose Dev Mode
    • Under Branch, choose spectacles-tutorial-more
    • Under Incremental Validation > Target Ref, enter spectacles-tutorial
    • Click Start

Validate SQL incrementally​

You can also validate SQL changes incrementally. This is especially beneficial because Spectacles will only test Explores that have changed, which often makes validation faster for large Looker instances.

First, let's introduce some SQL errors. On the spectacles-tutorial branch, introduce some errors to the sql field of some of your dimensions and commit the change.

Next, let's edit our Suite to perform incremental SQL validation in addition to Content Validation.

  1. In the top menu, click Suites

  2. Click the Suite named Incremental Validation

  3. Enable the SQL Validator

  4. Configure the SQL Validator section with the following options:

    • Only incremental errors: Check the box

  5. Click Update Suite

  6. On the Suites page, next to the Incremental Validation Suite, click Create Run

    • Under Workspace, choose Dev Mode
    • Under Branch, choose spectacles-tutorial
    • Click Start

Spectacles will start a new run using incremental SQL and Content Validation.

Spectacles should identify the errors associated with the LookML change you made, but shouldn't show you any additional, unrelated errors, even if your Production branch has errors.

Spectacles will mark any Explores where the SQL has not changed as Unmodified, indicating that those Explores were skipped and not tested.

Like the Content Validator, you can also change the target for incremental SQL validation.

Validate style incrementally​

Running the Style Validator incrementally is especially useful because the it often generates an overwhelming number of errors when first enabled on a project.

For example, you can turn on incremental mode for your Pull Request-triggered Suite to test only incoming LookML for style violations, instead of having to clean up your entire project to get a passing run.

In incremental mode, the Style Validator will run twice: once on a branch or commit, and once on a target reference (e.g. Looker production or another branch). The validator will only return errors that are unique to your branch or commit.

First, let's introduce some style violations. On the spectacles-tutorial branch, create a new dimension without a description.

If there's a lkmlstyle.yml file in your Looker project's repo, ensure it will test for the rule D301 (Visible dimension missing description). Check out the Style Validator section of the Validators tutorial if you're unsure how this file works.

Next, let's edit our Suite to perform incremental SQL validation in addition to Content Validation.

  1. In the top menu, click Suites

  2. Click the Suite named Incremental Validation

  3. Enable the Style Validator (you may need to set a Deploy Key on your repo in order to enable this validator)

  4. Configure the Style Validator section with the following options:

    • Only incremental errors: Check the box

  5. Click Update Suite

  6. On the Suites page, next to the Incremental Validation Suite, click Create Run

    • Under Workspace, choose Dev Mode
    • Under Branch, choose spectacles-tutorial
    • Click Start
info

How does Spectacles decide if a style violation is new?

Spectacles looks at each violation on your branch or commit and at each violation on the target ref (e.g. Looker production, another branch, etc.).

Spectacles will return a violation found on your branch of commit if it can't find a violation on the target ref that matches:

  • Rule code
  • Filepath
  • Lineage (e.g. orders_view.order_id_dimension)
  • Marked line of LookML

If you modify existing LookML with violations on your target ref in a way that alters any of the above characteristics, Spectacles will return a violation, even in incremental mode.

For example, the following modifications will cause an existing violation to be raised:

  • Renaming a file
  • Renaming a dimension, measure, view, etc.
  • Changing the content of the line of LookML with a violation