Starter judgments

Each starter is a complete judgment definition (its question, type, recipe and thresholds) written for a common question. Its paths are placeholders such as {subject}, which you fill with the paths of your own documents. Writing a context recipe shows how to preview one with dry_run and create it. This is version 1 of the catalogue. A placeholder that is optional can be left out, and then what it fills is left out too: a fields entry, a last_n entry, or an outcome rule.

Ticket triage: is it urgent?

For support queues that need the tickets that can’t wait at the top. It reads the ticket’s subject and first message, its latest three messages when the conversation is an array, and the customer’s plan if you have one. Its status, assignee and tags are left out, so changing them re-runs nothing. Read p or the urgent threshold to sort or alert. from_starter: ticket_triage.urgency

Placeholder	Takes		What it is	Example
`subject`	`state.…` or `attributes.…`	required	The ticket’s subject line.	`state.subject`
`body`	`state.…` or `attributes.…`	required	The ticket’s first message or description.	`state.body`
`messages`	`state.…` or `attributes.…`	optional	The conversation, an array of messages; the latest three are read.	`state.messages`
`plan`	`state.…` or `attributes.…`	optional	The customer’s plan or tier.	`attributes.plan`

The definition, with its placeholders:

{
  "name": "urgent",
  "type": "bool",
  "question": "Is this ticket urgent: does the customer need a reply within hours rather than days?",
  "criteria": "Urgent when a service is down or unusable, money is being lost or charged wrongly, data may be lost or exposed, or the customer faces a deadline today. A question, a feature request or a minor inconvenience is not urgent, however it is worded.",
  "context": {
    "fields": [
      "{subject}",
      "{body}",
      "{plan}"
    ],
    "last_n": {
      "{messages}": 3
    },
    "max_tokens": 1500
  },
  "thresholds": {
    "urgent": 0.8
  }
}

Created with the examples:

{
  "from_starter": "ticket_triage.urgency",
  "paths": {
    "subject": "state.subject",
    "body": "state.body",
    "messages": "state.messages",
    "plan": "attributes.plan"
  },
  "engine": {
    "name": "jev",
    "version": "current"
  }
}

Ticket triage: which team should take it?

For routing tickets to a team. It reads the same parts of a ticket as the urgency starter, so both go to the engine in one request, and picks billing, technical, account or sales, or none_of_the_above when none fits. Rename the options to your teams by editing the definition the dry run returns. Use value as the answer: a choice’s probabilities are overconfident until you post outcomes. from_starter: ticket_triage.routing

Placeholder	Takes		What it is	Example
`subject`	`state.…` or `attributes.…`	required	The ticket’s subject line.	`state.subject`
`body`	`state.…` or `attributes.…`	required	The ticket’s first message or description.	`state.body`
`messages`	`state.…` or `attributes.…`	optional	The conversation, an array of messages; the latest three are read.	`state.messages`
`plan`	`state.…` or `attributes.…`	optional	The customer’s plan or tier.	`attributes.plan`

The definition, with its placeholders:

{
  "name": "route",
  "type": "choice",
  "question": "Which team should handle this ticket?",
  "options": [
    {
      "value": "billing",
      "description": "Charges, invoices, refunds, payment methods and plan changes."
    },
    {
      "value": "technical",
      "description": "Bugs, errors, outages, performance and integrations."
    },
    {
      "value": "account",
      "description": "Signing in, access, users, permissions and account settings."
    },
    {
      "value": "sales",
      "description": "Pricing questions, upgrades, new purchases and contracts."
    }
  ],
  "context": {
    "fields": [
      "{subject}",
      "{body}",
      "{plan}"
    ],
    "last_n": {
      "{messages}": 3
    },
    "max_tokens": 1500
  }
}

Created with the examples:

{
  "from_starter": "ticket_triage.routing",
  "paths": {
    "subject": "state.subject",
    "body": "state.body",
    "messages": "state.messages",
    "plan": "attributes.plan"
  },
  "engine": {
    "name": "jev",
    "version": "current"
  }
}

Moderation: does this post break a rule?

For user-generated posts, comments and reviews. It reads the post’s text and title. The criteria list common community rules: replace them with your own by editing the definition the dry run returns. Likes, views and replies re-run nothing. Hide above the hide threshold and send the band between review and hide to a person. To show the post it replies to as well, add a relation; see the referenced document guide. from_starter: moderation.breaks_rule

Placeholder	Takes		What it is	Example
`text`	`state.…` or `attributes.…`	required	The post’s text.	`state.text`
`title`	`state.…` or `attributes.…`	optional	The post’s title, if posts have one.	`state.title`

The definition, with its placeholders:

{
  "name": "breaks_rule",
  "type": "bool",
  "question": "Does this post break any of the community's rules?",
  "criteria": "A post breaks a rule if it harasses or threatens a person, attacks people for who they are, is spam or a scam, shares someone's private information, or sexualises minors. Strong language, criticism, disagreement and off-topic posts do not break a rule on their own.",
  "context": {
    "fields": [
      "{title}",
      "{text}"
    ],
    "max_tokens": 1000
  },
  "thresholds": {
    "hide": 0.9,
    "review": 0.5
  }
}

Created with the examples:

{
  "from_starter": "moderation.breaks_rule",
  "paths": {
    "text": "state.text",
    "title": "state.title"
  },
  "engine": {
    "name": "jev",
    "version": "current"
  }
}

Churn risk: will this account cancel within 30 days?

For B2B and subscription accounts, where the signs are in the account’s tickets and invoices rather than the account itself. It judges only accounts, and reads each one’s name and plan, its latest eight tickets and a summary of its invoices from the last 180 days: how many, and the latest status. A new ticket or invoice re-judges the account. It predicts 30 days ahead, so an account that has not cancelled 30 days after an answer counts as a no, and, if you give the status path, an account whose status becomes cancelled counts as a yes: calibration learns from both without you posting anything. from_starter: churn_risk.at_risk

Placeholder	Takes		What it is	Example
`kind`	`attributes.…`	required	The attribute that says what kind of document each one is.	`attributes.kind`
`account_kind`	a value	required	Its value on accounts.	`account`
`ticket_kind`	a value	required	Its value on support tickets.	`ticket`
`invoice_kind`	a value	required	Its value on invoices.	`invoice`
`account_ref`	`attributes.…`	required	The attribute on tickets and invoices that holds their account’s document id.	`attributes.account_id`
`name`	`state.…` or `attributes.…`	required	The account’s name.	`state.name`
`plan`	`state.…` or `attributes.…`	optional	The account’s plan or tier.	`attributes.plan`
`ticket_subject`	`state.…` or `attributes.…`	required	A ticket’s subject line.	`state.subject`
`ticket_status`	`state.…` or `attributes.…`	required	A ticket’s status.	`state.status`
`invoice_status`	`state.…` or `attributes.…`	required	An invoice’s status, such as paid or overdue.	`state.status`
`status`	`state.…` or `attributes.…`	optional	The account’s status, for the outcome rule.	`attributes.status`
`cancelled`	a value	optional	The account status that means it cancelled.	`cancelled`

The definition, with its placeholders:

{
  "name": "churn_risk",
  "type": "bool",
  "question": "Will this account cancel within the next 30 days?",
  "criteria": "Signs of risk: repeated or unresolved tickets, frustration, mentions of cancelling, downgrading or a competitor, and failed or overdue invoices. A single routine question, or an account with paid invoices and quiet tickets, is not a sign.",
  "applies_to": {
    "{kind}": "{account_kind}"
  },
  "context": {
    "fields": [
      "{name}",
      "{plan}"
    ],
    "related": {
      "tickets": {
        "match": {
          "{kind}": "{ticket_kind}"
        },
        "join": {
          "theirs": "{account_ref}",
          "mine": "id"
        },
        "last_n": 8,
        "fields": [
          "created_at",
          "{ticket_subject}",
          "{ticket_status}"
        ]
      },
      "invoices": {
        "match": {
          "{kind}": "{invoice_kind}"
        },
        "join": {
          "theirs": "{account_ref}",
          "mine": "id"
        },
        "window": "180d",
        "aggregate": {
          "count": true,
          "latest": [
            "{invoice_status}"
          ]
        }
      }
    },
    "max_tokens": 2500
  },
  "horizon": "30d",
  "thresholds": {
    "at_risk": 0.6
  }
}

Suggested outcome setting, applied with the judgment’s first version (rules only on a plan with outcome rules):

{
  "implicit_negatives": true,
  "rules": [
    {
      "when": {
        "path": "{status}",
        "becomes": "{cancelled}"
      },
      "value": true
    }
  ]
}

Created with the examples:

{
  "from_starter": "churn_risk.at_risk",
  "paths": {
    "kind": "attributes.kind",
    "account_kind": "account",
    "ticket_kind": "ticket",
    "invoice_kind": "invoice",
    "account_ref": "attributes.account_id",
    "name": "state.name",
    "plan": "attributes.plan",
    "ticket_subject": "state.subject",
    "ticket_status": "state.status",
    "invoice_status": "state.status",
    "status": "attributes.status",
    "cancelled": "cancelled"
  },
  "engine": {
    "name": "jev",
    "version": "current"
  }
}

Lead quality: how good is this lead?

For inbound leads from forms, sign-ups or chats. It reads the company, the person’s role and what they wrote, and the company size and source if you have them, and scores the lead from 1 (not a fit) to 5 (ready to buy). Read score as a position on the scale rather than an exact level: the sales_ready threshold sits between levels 3 and 4. from_starter: lead_quality.score

Placeholder	Takes		What it is	Example
`company`	`state.…` or `attributes.…`	required	The lead’s company.	`state.company`
`role`	`state.…` or `attributes.…`	required	The person’s job title or role.	`state.title`
`message`	`state.…` or `attributes.…`	required	What they wrote: the form’s message or the chat.	`state.message`
`company_size`	`state.…` or `attributes.…`	optional	The company’s size, such as employees.	`attributes.company_size`
`source`	`state.…` or `attributes.…`	optional	Where the lead came from.	`attributes.source`

The definition, with its placeholders:

{
  "name": "lead_quality",
  "type": "score",
  "question": "How likely is this lead to become a paying customer?",
  "criteria": "Judge fit and intent together: a company that could use the product, a role that can buy or influence buying, and a message with a concrete need or timeline score high. Students, job seekers, competitors, vendors and spam score lowest.",
  "levels": [
    {
      "value": 1,
      "label": "not a fit",
      "description": "Spam, a student, a job seeker, a vendor or a competitor."
    },
    {
      "value": 2,
      "label": "weak",
      "description": "A possible fit with no stated need."
    },
    {
      "value": 3,
      "label": "possible",
      "description": "A fit with a vague need or no timeline."
    },
    {
      "value": 4,
      "label": "good",
      "description": "A fit with a concrete need."
    },
    {
      "value": 5,
      "label": "ready to buy",
      "description": "A fit with a concrete need, a timeline or budget, and a buyer."
    }
  ],
  "context": {
    "fields": [
      "{company}",
      "{role}",
      "{message}",
      "{company_size}",
      "{source}"
    ],
    "max_tokens": 1000
  },
  "thresholds": {
    "sales_ready": 3.5
  }
}

Created with the examples:

{
  "from_starter": "lead_quality.score",
  "paths": {
    "company": "state.company",
    "role": "state.title",
    "message": "state.message",
    "company_size": "attributes.company_size",
    "source": "attributes.source"
  },
  "engine": {
    "name": "jev",
    "version": "current"
  }
}

Review queue: does a person need to look at this?

For any flow where most items can be handled automatically and some need a person: refund requests, automated replies, applications, claims. It reads the item’s content and, if you have them, its summary and category. Items above the review threshold go to the queue; lower the threshold to review more. from_starter: review_queue.needs_review

Placeholder	Takes		What it is	Example
`content`	`state.…` or `attributes.…`	required	The item itself: the request, reply or application.	`state.content`
`summary`	`state.…` or `attributes.…`	optional	A short summary or title.	`state.summary`
`category`	`state.…` or `attributes.…`	optional	The item’s category or type.	`attributes.category`

The definition, with its placeholders:

{
  "name": "needs_review",
  "type": "bool",
  "question": "Should a person review this item before it is acted on?",
  "criteria": "Review when the item is ambiguous or contradicts itself, involves money, legal matters, safety or access to an account, the person is upset, or an automated answer could easily be wrong. Routine, clear requests do not need review.",
  "context": {
    "fields": [
      "{summary}",
      "{category}",
      "{content}"
    ],
    "max_tokens": 1500
  },
  "thresholds": {
    "review": 0.5
  }
}

Created with the examples:

{
  "from_starter": "review_queue.needs_review",
  "paths": {
    "content": "state.content",
    "summary": "state.summary",
    "category": "attributes.category"
  },
  "engine": {
    "name": "jev",
    "version": "current"
  }
}

Get started

Before you build

Concepts

Guides

Engines

Reference

Starter judgments

Ticket triage: is it urgent?

Ticket triage: which team should take it?

Moderation: does this post break a rule?

Churn risk: will this account cancel within 30 days?

Lead quality: how good is this lead?

Review queue: does a person need to look at this?

​Ticket triage: is it urgent?

​Ticket triage: which team should take it?

​Moderation: does this post break a rule?

​Churn risk: will this account cancel within 30 days?

​Lead quality: how good is this lead?

​Review queue: does a person need to look at this?

Ticket triage: is it urgent?

Ticket triage: which team should take it?

Moderation: does this post break a rule?

Churn risk: will this account cancel within 30 days?

Lead quality: how good is this lead?

Review queue: does a person need to look at this?