Packages
A matching language for matching queries against token-based natural language input. Like Regular Expressions, but for for natural language.
Current section
47 Versions
Jump to
Current section
47 Versions
Compare versions
7
files changed
+150
additions
-61
deletions
| @@ -62,11 +62,15 @@ rules. Each individual has the following syntax: | |
| 62 62 | |
| 63 63 | Basic words; rules consisting of only alphanumeric characters. |
| 64 64 | |
| 65 | - Matching is done on both the lowercased, normalized version of the |
| 66 | - word, and on the lemmatization of the word. |
| 65 | + Matching is done on both the lowercased, normalized, accents-removed |
| 66 | + version of the word, and on the lemmatization of the word. The *lemma* |
| 67 | + of a word is its base version; e.g. for verbs it is the root form (are |
| 68 | + → be, went → go); for nouns it is the singular form of the word. |
| 67 69 | |
| 68 | - Use a dash (`-`) to match on compound nouns: `was-machine` matches |
| 69 | - all of `wasmachine`, `was-machine` and `was machine`. |
| 70 | + Some languages (german, dutch, …) have compound nouns, that are often |
| 71 | + written both with and without spaces or dashes. Use a dash (`-`) to |
| 72 | + match on such compound nouns: the rule `was-machine` matches all of |
| 73 | + `wasmachine`, `was-machine` and `was machine`. |
| 70 74 | |
| 71 75 | |
| 72 76 | ### Literals |
| @@ -74,7 +78,8 @@ all of `wasmachine`, `was-machine` and `was machine`. | |
| 74 78 | `"Literal word sequence"` |
| 75 79 | |
| 76 80 | Matches a literal piece of text, which can span multiple |
| 77 | - tokens. Matching is **case insensitive**. |
| 81 | + tokens. Matching is **case insensitive**, and also insensitive to |
| 82 | + the presence of accented characters. |
| 78 83 | |
| 79 84 | |
| 80 85 | ### Ignoring tokens: _ |
| @@ -82,10 +87,11 @@ tokens. Matching is **case insensitive**. | |
| 82 87 | `hello _ world` |
| 83 88 | |
| 84 89 | The standalone occurence of `_` matches 0-5 of any available token, |
| 85 | - greedy. |
| 90 | + non-greedy. This can be used in places where you expect a few tokens |
| 91 | + to occur but you don't care about the tokens. |
| 86 92 | |
| 87 93 | |
| 88 | - ### Stand-alone range specifiers |
| 94 | + ### Matching a range of tokens |
| 89 95 | |
| 90 96 | - `[1]` match exactly one token; any token |
| 91 97 | - `[2+]` match 2 or more tokens (greedy) |
| @@ -93,6 +99,9 @@ greedy. | |
| 93 99 | - `[2+?]` match 2 or more tokens (non-greedy) |
| 94 100 | - `[1-3?]` match 1 to 3 tokens (non-greedy) |
| 95 101 | |
| 102 | + Use this when you know how many tokens you need to match, but it does |
| 103 | + not matter what the contents of the tokens is. |
| 104 | + |
| 96 105 | |
| 97 106 | ### Entities |
| 98 107 | |
| @@ -103,6 +112,42 @@ e.g. by an NLP NER engine like Duckling. | |
| 103 112 | Entities are automatically captured under a variable with the same |
| 104 113 | name as the entity's kind. |
| 105 114 | |
| 115 | + The default list of supported entities is the following: |
| 116 | + |
| 117 | + - `amount_of_money` (duckling) |
| 118 | + - `credit_card_number` (duckling) |
| 119 | + - `date` (spacy) |
| 120 | + - `distance` (duckling) |
| 121 | + - `duration` (duckling) |
| 122 | + - `email` (duckling) |
| 123 | + - `event` (spacy) |
| 124 | + - `fac` (spacy) |
| 125 | + - `gpe` (spacy) |
| 126 | + - `language` (spacy) |
| 127 | + - `law` (spacy) |
| 128 | + - `loc` (spacy) |
| 129 | + - `money` (spacy) |
| 130 | + - `norp` (spacy) |
| 131 | + - `number` (duckling) |
| 132 | + - `ordinal` (duckling) |
| 133 | + - `org` (spacy) |
| 134 | + - `percent` (spacy) |
| 135 | + - `person` (spacy) |
| 136 | + - `phone_number` (duckling) |
| 137 | + - `product` (spacy) |
| 138 | + - `quantity` (duckling) |
| 139 | + - `temperature` (duckling) |
| 140 | + - `time` (duckling) |
| 141 | + - `url` (duckling) |
| 142 | + - `volume` (duckling) |
| 143 | + - `work_of_art` (spacy) |
| 144 | + |
| 145 | + From our experience, Duckling entities work much better than Spacy |
| 146 | + entities, and are preferred for use. Besides being more accurate, the |
| 147 | + Duckling entities also provide more metadata, like valid UTC times |
| 148 | + when a date is recognized. |
| 149 | + |
| 150 | + |
| 106 151 | |
| 107 152 | ### Regular expressions |
| 108 153 | |
| @@ -119,6 +164,8 @@ KL12345 and extracts `12345` as the `flight_number` capture. | |
| 119 164 | |
| 120 165 | ### OR / grouping construct |
| 121 166 | |
| 167 | + Use parentheses combined with the pipe `|` character to specify an OR clause. |
| 168 | + |
| 122 169 | - `pizza | fries | chicken` - OR-clause on the root level without |
| 123 170 | parens, matches either token |
| 124 171 | |
| @@ -126,43 +173,86 @@ KL12345 and extracts `12345` as the `flight_number` capture. | |
| 126 173 | matches one token consisting of first `a`, and then `a`, `b` or |
| 127 174 | `c`. |
| 128 175 | |
| 129 | - - `( a )[3+]` matches 3 or more token consisting of `a` |
| 130 176 | - `( hi | hello )[=greeting]` matches 1 token and stores it in `greeting` |
| 131 177 | |
| 178 | + Parenthesis with | can also be used to capture a sequence of tokens together as one group: |
| 179 | + |
| 180 | + - `( a )[3+]` matches 3 or more token consisting of `a` |
| 181 | + |
| 132 182 | |
| 133 183 | ### Permutation construct |
| 134 184 | |
| 135 | - - `< a b c >` matches any permutation of the sequence `a b c`; `a c b`, or `b a c`, or `c a b`, etc |
| 185 | + The permutation construct using pointy brackets, `<`, `>` matches the |
| 186 | + given rules in no particular order. |
| 187 | + |
| 188 | + `< a b c >` matches any permutation of the sequence `a b c`; `a c b`, or `b a c`, or `c a b`, etc |
| 189 | + |
| 190 | + An implicit `_` is inserted between all rules. So the rule `<a b>` can |
| 191 | + also be written as `(a _ b | b _ a)`. |
| 136 192 | |
| 137 193 | |
| 138 194 | ### Start / end sentence markers |
| 139 195 | |
| 196 | + To match the beginning of end of sentences, the following constructs can be used: |
| 197 | + |
| 140 198 | - `[Start]` Matches the start of a sentence |
| 141 199 | - `[End]` Matches the end of a sentence |
| 142 200 | |
| 201 | + > The `[Start]` and `[End]` symbols are not always the same as the |
| 202 | + > start and end of the input string, as sometimes the user input is |
| 203 | + > split into multiple sentences, based on the Spacy sentence |
| 204 | + > tokenizer. |
| 143 205 | |
| 144 | - ### Word collections ("concepts") |
| 145 | - |
| 146 | - - `@food` matches any token in the `food` collection. |
| 147 | - - `@food.subcat` matches any token in the given subcategory. |
| 148 | - |
| 149 | - Concept compilation is done as part of the parse phase; the concepts |
| 150 | - compiler must must return an `{m, f, a}` triple. In runtime, this MFA |
| 151 | - is called while matching, and thus, it must be a fast function. |
| 152 206 | |
| 153 207 | ### Part-of-speech tags (word kinds) |
| 154 208 | |
| 209 | + To be able to disambiguate between word kinds, the `%` construct |
| 210 | + matches on the POS-tag of a token: |
| 211 | + |
| 155 212 | - `%VERB` matches any verb |
| 156 213 | - `%NOUN` matches any noun |
| 157 | - - Any other POS Spacy tags are valid as well |
| 214 | + |
| 215 | + Any other [POS Spacy tags](https://spacy.io/api/annotation#pos-en) are |
| 216 | + valid as well. |
| 158 217 | |
| 159 218 | |
| 160 | - ### Rule modifiers |
| 219 | + ### Optionality modifier |
| 220 | + |
| 221 | + An appended `?` makes the given rule optional (it needs to occur 0 or 1 times). |
| 222 | + |
| 223 | + |
| 224 | + ### Repetition modifier |
| 161 225 | |
| 162 226 | Any rule can have a `[]` block which contains a repetition modifier |
| 163 227 | and/or a capture expression. |
| 164 228 | |
| 165 | - Entity blocks are automatically captured as the entity kind. |
| 229 | + - `a[1]` match exactly one `a` word |
| 230 | + - `a[2+]` match 2 or more `a`'s (greedy) |
| 231 | + - `a[1-3]` match 1 to 3 `a`'s (greedy) |
| 232 | + - `a[2+?]` match 2 or more `a`'s (non-greedy) |
| 233 | + - `a[1-3?]` match 1 to 3 `a`'s (non-greedy) |
| 234 | + |
| 235 | + |
| 236 | + ### Capture modifier |
| 237 | + |
| 238 | + `(my name is _)[=x]` stores the entire token sequence "My name is john" |
| 239 | + |
| 240 | + |
| 241 | + ### Punctuation |
| 242 | + |
| 243 | + Punctuation is optional, and can be ignored while creating match |
| 244 | + rules. However, punctuation tokens *are* stored in the tokenized |
| 245 | + version of the input; in fact, multiple *tokenizations* of the input |
| 246 | + are stored for each sentence, one without and one with with the |
| 247 | + punctuation. |
| 248 | + |
| 249 | + The sentence `Hello, world.` is stored both as: |
| 250 | + |
| 251 | + - `Hello` `world` |
| 252 | + - `Hello` `,` `world` `.` |
| 253 | + |
| 254 | + Matching punctuation can be done by including the punctuation into `'` |
| 255 | + literal quotes. |
| 166 256 | |
| 167 257 | |
| 168 258 | ## Sentence tokenization |
| @@ -1 +1 @@ | |
| 1 | - 0.3.5 |
| \ No newline at end of file | |
| 1 | + 0.3.6 |
| \ No newline at end of file |
| @@ -41,4 +41,4 @@ | |
| 41 41 | {<<"optional">>,false}, |
| 42 42 | {<<"repository">>,<<"hexpm">>}, |
| 43 43 | {<<"requirement">>,<<"~> 0.1.1">>}]]}. |
| 44 | - {<<"version">>,<<"0.3.5">>}. |
| 44 | + {<<"version">>,<<"0.3.6">>}. |
| @@ -142,29 +142,27 @@ defmodule BubbleMatch.Matcher do | |
| 142 142 | end |
| 143 143 | |
| 144 144 | defp match_rule({:regex, re, _}, _rls_remaining, [t | _] = ts_remaining, context) do |
| 145 | - with :nomatch <- boolean_match(&Token.regex?(&1, re), ts_remaining, context) do |
| 146 | - input_str = |
| 147 | - case ts_remaining do |
| 148 | - [%{index: 0} | _] -> Enum.map(ts_remaining, & &1.raw) |
| 149 | - _ -> [" " | Enum.map(ts_remaining, & &1.raw)] |
| 150 | - end |
| 151 | - |> IO.chardata_to_string() |
| 152 | - |
| 153 | - case Regex.scan(re, input_str) do |
| 154 | - [[capture | groups] | _] -> |
| 155 | - [before, _] = String.split(input_str, capture, parts: 2) |
| 156 | - |
| 157 | - start_idx = t.start + String.length(before) |
| 158 | - {_ignore, rest} = Enum.split_with(ts_remaining, &(&1.end < start_idx)) |
| 159 | - end_idx = start_idx + String.length(capture) |
| 160 | - {ts_match, ts_remaining} = Enum.split_with(rest, &(&1.start <= end_idx)) |
| 161 | - |
| 162 | - context = opt_add_regex_captures(groups, context, re, input_str) |
| 163 | - {:match, ts_remaining, Enum.reverse(ts_match), context} |
| 164 | - |
| 165 | - [] -> |
| 166 | - :nomatch |
| 145 | + input_str = |
| 146 | + case ts_remaining do |
| 147 | + [%{index: 0} | _] -> Enum.map(ts_remaining, & &1.raw) |
| 148 | + _ -> [" " | Enum.map(ts_remaining, & &1.raw)] |
| 167 149 | end |
| 150 | + |> IO.chardata_to_string() |
| 151 | + |
| 152 | + case Regex.scan(re, input_str) do |
| 153 | + [[capture | groups] | _] -> |
| 154 | + [before, _] = String.split(input_str, capture, parts: 2) |
| 155 | + |
| 156 | + start_idx = t.start + String.length(before) |
| 157 | + {_ignore, rest} = Enum.split_with(ts_remaining, &(&1.end < start_idx)) |
| 158 | + end_idx = start_idx + String.length(capture) |
| 159 | + {ts_match, ts_remaining} = Enum.split_with(rest, &(&1.start <= end_idx)) |
| 160 | + |
| 161 | + context = opt_add_regex_captures(groups, context, re, input_str) |
| 162 | + {:match, ts_remaining, Enum.reverse(ts_match), context} |
| 163 | + |
| 164 | + [] -> |
| 165 | + :nomatch |
| 168 166 | end |
| 169 167 | end |
| 170 168 | |
| @@ -191,7 +189,7 @@ defmodule BubbleMatch.Matcher do | |
| 191 189 | {[], ts_remaining, str}, |
| 192 190 | fn |
| 193 191 | t, {matched, remaining, str} -> |
| 194 | - raw = String.trim_trailing(t.raw) |> String.downcase() |
| 192 | + raw = String.trim_trailing(t.raw) |> Token.base_form() |
| 195 193 | |
| 196 194 | if String.starts_with?(str, raw) do |
| 197 195 | case String.trim_leading(str, raw) do |
| @@ -4,6 +4,7 @@ defmodule BubbleMatch.Parser do | |
| 4 4 | import NimbleParsec |
| 5 5 | |
| 6 6 | alias BubbleMatch.ParseError |
| 7 | + alias BubbleMatch.Token |
| 7 8 | |
| 8 9 | @ws [9, 10, 11, 12, 13, 32] |
| 9 10 | ws = ignore(utf8_string(@ws, min: 1)) |
| @@ -23,7 +24,7 @@ defmodule BubbleMatch.Parser do | |
| 23 24 | int = times(utf8_char([?0..?9]), min: 1) |> reduce(:to_int) |
| 24 25 | |
| 25 26 | defp finalize_literal([word]) do |
| 26 | - {:literal, String.downcase(word)} |
| 27 | + {:literal, Token.base_form(word)} |
| 27 28 | end |
| 28 29 | |
| 29 30 | literal = fn char -> |
| @@ -139,13 +140,12 @@ defmodule BubbleMatch.Parser do | |
| 139 140 | |> ignore(string("]")) |
| 140 141 | |
| 141 142 | defp finalize_word([str]) do |
| 142 | - word = String.downcase(str) |> Unidekode.to_ascii() |
| 143 | - {:word, word} |
| 143 | + {:word, Token.base_form(str)} |
| 144 144 | end |
| 145 145 | |
| 146 146 | defp finalize_word([a, b]) do |
| 147 | - a = String.downcase(a) |
| 148 | - b = String.downcase(b) |
| 147 | + a = Token.base_form(a) |
| 148 | + b = Token.base_form(b) |
| 149 149 | {:or, [[{:word, a <> b, []}], [{:word, a <> "-" <> b, []}], [{:word, a, []}, {:word, b, []}]]} |
| 150 150 | end |
| 151 151 | |
| @@ -400,10 +400,10 @@ defmodule BubbleMatch.Parser do | |
| 400 400 | rules, |
| 401 401 | fn |
| 402 402 | {:entity, kind, meta} -> |
| 403 | - {:entity, kind, Keyword.put(meta, :assign, meta[:assign] || String.downcase(kind))} |
| 403 | + {:entity, kind, Keyword.put(meta, :assign, meta[:assign] || Token.base_form(kind))} |
| 404 404 | |
| 405 405 | {:concept, {kind}, meta} -> |
| 406 | - {:concept, {kind}, Keyword.put(meta, :assign, meta[:assign] || String.downcase(kind))} |
| 406 | + {:concept, {kind}, Keyword.put(meta, :assign, meta[:assign] || Token.base_form(kind))} |
| 407 407 | |
| 408 408 | {verb, rules, meta} when is_list(rules) -> |
| 409 409 | {verb, Enum.map(rules, &add_implicit_assign/1), meta} |
Loading more files…