About CLEF

Pattern Languages

The catalogue uses three pattern languages, matching different kinds of detection:

Regex (Regular expressions)

For surface-level patterns — matching character sequences in word forms without requiring any NLP pipeline. Used especially for derivational morphology features (prefixes and suffixes) where the orthographic form is the signal.

\w+tion$                                   words ending in -tion
^un\w+                                     words starting with un-
\w+'(t|ll|re|ve|s|d)                       contractions

CQL (Corpus Query Language)

For sequential patterns — matching tokens by POS tag, UPOS, lemma, word form, or dependency relation. The dominant query language in corpus linguistics (CQP/CWB, Sketch Engine, NoSketch Engine).

[pos="JJ"] [pos="NN|NNS"]                  two-token sequence
[lemma="be" & pos="VBZ"]                   lemma AND POS constraint
[word={words} & pos="RB"]                  word list with placeholder
[pos="IN" & dep="prep"]                    POS + dependency relation
[upos="VERB" & pos!="VBG"]                 coarse UPOS + fine-grained exclusion
[dep="aux.*"] [upos="ADV"] [upos="VERB"]   regex in dep, UPOS constraints

Semgrex (Stanford)

For structural patterns — matching dependency tree configurations. The de facto standard for dependency patterns.

{pos:/VB.*/}=verb >auxpass {lemma:be}      passive voice
{pos:/NN.*/}=noun >amod {}=adj             attributive adjective
A >rel B                                   A is head of B via relation

Rule Fields

Each detection rule is a YAML mapping with the following fields:

Field	Required	Description
source		Provenance key (e.g. pybiber, mfte). Rules without a source are universal.
requires		List of token attributes needed: [word, pos, dep]
cql		CQL pattern — sequential token matching by POS, lemma, word, dep, etc.
semgrex		Semgrex pattern — dependency tree matching
regex		Regular expression — surface-level character matching
words		Word list (flat list or named dict of lists)
parts		Named sub-patterns — a dict of parts, each with cql:, semgrex:, or word_list:
combine		Boolean expression composing parts and feature codes: p1 | p2, _ & !FEAT, _ & -FEAT
anchor		Which token in a multi-token pattern to report as the hit: first (default) or last
sentence_scope		If true, pattern matching is constrained to within sentence boundaries
refines		POS tag(s) this rule reclassifies (e.g. RB). See Tag Refinement below.
s_attrs		Sentence-level attribute specs — mapping of names to CQL patterns. Enables [sent_has_q="true"] style constraints.
description		Human-readable explanation of the rule
caveat		Known limitation or accuracy warning
usas_code		USAS semantic tag code (for semantic features)

Combine Expressions

The combine field uses boolean expressions to compose parts and exclude other features. Two NOT operators handle different kinds of feature overlap:

! exclude (index overlap)

Removes hits whose token index also appears in the excluded feature. Use when both features match the same token — e.g. _ & !PASSBY on PASS.

- subtract (count)

Subtracts the total count of another feature from this feature's count. Use when features overlap conceptually but match different tokens — e.g. _ & -WHREL_SUBJ & -WHREL_OBJ on WHSC.

| union (parts)

Combines hits from multiple named parts — e.g. p1 | p2 | p3 for multi-pattern rules.

_ self

Refers to the rule's own bare pattern. Used when the rule has a cql: or semgrex: at rule level alongside a combine expression — e.g. _ & !NOMZ & !GER.

Sentence-Level Attributes (s-attrs)

Some features require checking a property of the sentence a token belongs to, not just the token's immediate context. For example, WH-questions are identified partly by whether the sentence contains a question mark or an auxiliary verb — properties that may be many tokens away from the WH-word itself.

CLEF handles this with s-attributes: pre-computed sentence-level boolean properties attached to every token in the sentence. A detection rule declares the properties it needs as CQL patterns; the engine evaluates each pattern per sentence and sets the named attribute to "true" on every token in matching sentences. CQL constraints then reference these attributes like any other token property.

# Rule declaration
s_attrs:
  sent_has_q: '[word="?"]'           sentence contains "?"
  sent_has_aux: '[upos="AUX"]'       sentence contains an auxiliary

# CQL pattern referencing s-attributes
[pos="W.*" & sent_has_q="true"]     WH-word in a question sentence

This mechanism is general: any sentence-level property expressible as a CQL pattern can be declared and queried, without hard-coding attribute names in the engine.

Composites and Atomics

When a source (e.g. MFTE) splits a Biber feature, the catalogue preserves both levels. A composite feature has children listing its atomic parts; each child has a parent back-link. For example, Biber’s “first person pronouns” (6) becomes composite FPP with children FPP1S and FPP1P.

Tag Refinement

Some detection rules reclassify tokens that a POS tagger has assigned to a broad category. For example, a detection rule for FREQ (frequency adverbs) declares refines: RB: tokens it matches are functionally "frequency adverbs," not generic adverbs. The residual RB count should only include adverbs that no specific detection rule has claimed.

CLEF handles this with two virtual layers:

pos — the POS tagger's output (immutable)

Always reflects what the tagger assigned. A query like [pos="RB"] matches all RB tokens, including those later refined.

cat — functional category (refined)

Defaults to pos but is overwritten when a detection rule with refines: RB matches. A query like [cat="RB"] matches only unrefined RB tokens — those not claimed by any specific detection rule.

Detection rules declare refinement with the refines field (e.g. refines: RB or refines: RB|NN for multiple POS tags). This makes explicit a mechanism that in other tools is implicit in processing order — see the design principle of separation of spec and implementation above.