Overview

Some tests are not currently possible to create via the "Create custom test" UI: for example, Vanta does not support creating tests on multiple resources at once via the UI alone. You can create a custom test that involves multiple resources, however, by computing the test logic yourself and sending the test outcomes as boolean properties on a Custom Resource.

Steps

Follow the My First Connector Guide to create a private integration and a custom resource with a boolean property.
Follow the GraphQL API Quick start to create an API token.
Query the API for the data that you need to compute the test.
Compute the test outcome using your language of choice for each resource.
Send the outcomes as boolean properties on custom resources.
Create a custom test on the boolean property being true or false.

Example

The "Employee computers are monitored with the Vanta Agent or an MDM" test makes sure that employee computers have an Agent or MDM installed, but you may want to ensure certain employees within certain departments have a particular MDM installed. For example, you may want to make sure that everyone in the "Legal" department has MicrosoftEndpointManager installed, since they use Windows computers. The employee department is determined by your HR system, Bob, which is integrated with Vanta. This is currently impossible to do via the "Create custom test" UI alone, since this test depends on data from two different resources: BobHrUsers and MicrosoftEndpointManagerManagedComputers.

Steps

First, create a private integration. Then, create a Custom Resource called LegalEmployee with a boolean property hasMEMInstalled. Here is the JTD Schema for the custom properties:

{
  "properties": {
    "hasMEMInstalled": {
      "type": "boolean"
    }
  }
}

Then, create an API token for the GraphQL API.

Next, query the relevant data from the GraphQL API:

// Using the graphql-request library (npm install graphql graphql-request)
import { GraphQLClient, gql } from "graphql-request";

const endpoint = "https://api.vanta.com/graphql";

// Initialize client
const graphQLClient = new GraphQLClient(endpoint, {
  headers: {
    Authorization: "token YOUR_TOKEN_HERE",
  },
});

// Set up query
const BobHrUsersQuery = gql`
  query getBobHrUsers($first: Int!, $after: String) {
    organization {
        BobHrUserList(first: $first, after: $after) {
            totalCount
            edges {
                node {
                  department
                  vantaOwner {
                    uid
                  }
                }
            }
            pageInfo {
                startCursor
                hasPreviousPage
                hasNextPage
                endCursor
            }
        }
    }
  }
`;

const MEMComputersQuery = gql`
  query getMEMComputers($first: Int!, $after: String) {
    organization {
      MicrosoftEndpointManagerManagedComputerList(first: $first, after: $after) {
        totalCount
        edges {
          node {
            vantaOwner {
              uid
            }
          }
        }
        pageInfo {
          startCursor
          hasPreviousPage
          hasNextPage
          endCursor
        }
      }
    }
  }    
`

const getAllUsers = async () => {
  let users = [];
  let after = null;
  while (true) {
    const data = await graphQLClient.request(BobHrUsersQuery, { first: 100, after });
    users.push(...data.organization.BobHrUserList.edges.map(e => e.node));
    if (!data.organization.BobHrUserList.pageInfo.hasNextPage) {
      break;
    }
    after = data.organization.BobHrUserList.pageInfo.endCursor;
  }
  return users
}

const getAllMEMComputers = async () => {
  let users = [];
  let after = null;
  while (true) {
    const data = await graphQLClient.request(MEMComputersQuery, { first: 100, after });
    users.push(...data.organization.BobHrUserList.edges.map(e => e.node));
    if (!data.organization.BobHrUserList.pageInfo.hasNextPage) {
      break;
    }
    after = data.organization.BobHrUserList.pageInfo.endCursor;
  }
  return users
}

Then, compute the test result and send the results via the custom resource's boolean property:

void (async () => {
  const legalHrUsers = 
    (await getAllUsers()).filter(u => u.department === "Legal");
  const userIdToMemComputer = new Map((await getAllMEMComputers()).map(c => [c.vantaOwner?.uid, c]));
  const testResults = legalHrUsers.map(u => {
    const uid = u.vantaOwner?.uid;
    if (uid === undefined) {
      throw new Error("uid should be defined")
    }

    const memComputer = userIdToMemComputer.get(uid);
    return {
      displayName: u.displayName,
      uniqueId: uid,
      externalUrl: u.externalUrl,
      customProperties: {
        hasMEMInstalled: memComputer !== undefined
      },
    }
  });
  await fetch("https://api.vanta.com/v1/respources/custom_resource/sync_all", {
    method: "PUT",
    headers: {
      "Authorization": "Bearer ACCESS_TOKEN_HERE",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      resourceId: "PRIVATE_RESOURCE_ID_HERE",
      resources: testResults,
    })
  })
})();

Run this script on some cadence to keep the test results up to date. For example, you could run this script in AWS Lambda or via Github Actions.