| <!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="Asciidoctor 2.0.26"><meta name="description" content="Discussion of Unicode and UTF-8/UTF-16/UTF-32 support"><title>pcre2unicode(3)</title><link rel="stylesheet" href="./css/document.css"><link rel="stylesheet" href="./css/standalone.css"><meta name="color-scheme" content="light dark"><meta name="theme-color" media="(prefers-color-scheme: light)" content="#0085cc"><meta name="theme-color" media="(prefers-color-scheme: dark)" content="#33b8ff"></head><body class="manpage toc2 toc-right offline-manual"><header class="offline-nav"><nav><a class="offline-index" href="./index.html">Index</a><a class="offline-home" href="https://www.pcre2.org/"><span aria-hidden="true">🌐︎</span> PCRE2 homepage</a></nav></header><main class="document-page"><header class="document-title"><h1>pcre2unicode(3)</h1><p class="document-meta">PCRE2 10.50-DEV manual; updated <time datetime="2024-11-27">27 Nov 2024</time></p></header><div class="document-layout"><nav id="toc" class="toc2" role="doc-toc"><h2 id="toc-title">Contents</h2><ol class="toc-list level-1"><li><a href="#_unicode_and_utf_support" title="Unicode and UTF support">Unicode and UTF support</a></li><li><a href="#_unicode_property_support" title="Unicode property support">Unicode property support</a></li><li><a href="#_wide_characters_and_utf_modes" title="Wide characters and UTF modes">Wide characters and UTF modes</a></li><li><a href="#_unicode_case_equivalence" title="Unicode case-equivalence">Unicode case-equivalence</a></li><li><a href="#script-runs" title="Script runs">Script runs</a><ol class="toc-list level-2"><li><a href="#basic-script-run-rules" title="Basic script run rules">Basic script run rules</a></li><li><a href="#_the_chinese_han_script" title="The Chinese Han script">The Chinese Han script</a></li><li><a href="#_decimal_digits" title="Decimal digits">Decimal digits</a></li></ol></li><li><a href="#_validity_of_utf_strings" title="Validity of UTF strings">Validity of UTF strings</a><ol class="toc-list level-2"><li><a href="#utf8-string-errors" title="Errors in UTF-8 strings">Errors in UTF-8 strings</a></li><li><a href="#_errors_in_utf_16_strings" title="Errors in UTF-16 strings">Errors in UTF-16 strings</a></li><li><a href="#_errors_in_utf_32_strings" title="Errors in UTF-32 strings">Errors in UTF-32 strings</a></li></ol></li><li><a href="#_matching_in_invalid_utf_strings" title="Matching in invalid UTF strings">Matching in invalid UTF strings</a></li></ol></nav><article class="prose" id="content"><section class="doc-section level-1"><h2 id="_unicode_and_utf_support">Unicode and UTF support</h2><p>PCRE2 is normally built with Unicode support, though if you do not need it, you | |
| can build it without, in which case the library will be smaller. With Unicode | |
| support, PCRE2 has knowledge of Unicode character properties and can process | |
| strings of text in UTF-8, UTF-16, and UTF-32 format (depending on the code unit | |
| width), but this is not the default. Unless specifically requested, PCRE2 | |
| treats each code unit in a string as one character.</p> | |
| <p>There are two ways of telling PCRE2 to switch to UTF mode, where characters may | |
| consist of more than one code unit and the range of values is constrained. The | |
| program can call | |
| <code class="function pcre2-api rouge"><span class="nf">pcre2_compile()</span></code> | |
| with the <code class="constant pcre2-api rouge"><span class="no">PCRE2_UTF</span></code> option, or the pattern may start with the sequence <code class="regex rouge"><span class="pcre2-semantic">(*UTF)</span></code>. | |
| However, the latter facility can be locked out by the <code class="constant pcre2-api rouge"><span class="no">PCRE2_NEVER_UTF</span></code> option. | |
| That is, the programmer can prevent the supplier of the pattern from switching | |
| to UTF mode.</p> | |
| <p>Note that the <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> option (see | |
| below) | |
| forces <code class="constant pcre2-api rouge"><span class="no">PCRE2_UTF</span></code> to be set.</p> | |
| <p>In UTF mode, both the pattern and any subject strings that are matched against | |
| it are treated as UTF strings instead of strings of individual one-code-unit | |
| characters. There are also some other changes to the way characters are | |
| handled, as documented below.</p></section> | |
| <section class="doc-section level-1"><h2 id="_unicode_property_support">Unicode property support</h2><p>When PCRE2 is built with Unicode support, the escape sequences <code class="regex rouge"><span class="pcre2-semantic">\p{..}</span></code>, | |
| <code class="regex rouge"><span class="pcre2-semantic">\P{..}</span></code>, and <code class="regex rouge"><span class="pcre2-semantic">\X</span></code> can be used. This is not dependent on the <code class="constant pcre2-api rouge"><span class="no">PCRE2_UTF</span></code> setting. | |
| The Unicode properties that can be tested are a subset of those that Perl | |
| supports. Currently they are limited to the general category properties such as | |
| <code class="regex rouge"><span class="pcre2-semantic">Lu</span></code> for an upper case letter or <code class="regex rouge"><span class="pcre2-semantic">Nd</span></code> for a decimal number, the derived properties | |
| <code class="regex rouge"><span class="pcre2-semantic">Any</span></code> and <code class="regex rouge"><span class="pcre2-semantic">Lc</span></code> (synonym <code class="regex rouge"><span class="pcre2-semantic">L&</span></code>), the Unicode script names such as <code class="regex rouge"><span class="pcre2-semantic">Arabic</span></code> or <code class="regex rouge"><span class="pcre2-semantic">Han</span></code>, | |
| <code class="regex rouge"><span class="pcre2-semantic">Bidi_Class</span></code>, <code class="regex rouge"><span class="pcre2-semantic">Bidi_Control</span></code>, and a few binary properties.</p> | |
| <p>The full lists are given in the | |
| <a href="./pcre2pattern.html"><code>pcre2pattern</code></a> | |
| and | |
| <a href="./pcre2syntax.html"><code>pcre2syntax</code></a> | |
| documentation. In general, only the short names for properties are supported. | |
| For example, <code class="regex rouge"><span class="pcre2-semantic">\p{L}</span></code> matches a letter. Its longer synonym, <code class="regex rouge"><span class="pcre2-semantic">\p{Letter}</span></code>, is not | |
| supported. Furthermore, in Perl, many properties may optionally be prefixed by | |
| <code class="regex rouge"><span class="pcre2-semantic">Is</span></code>, for compatibility with Perl 5.6. PCRE2 does not support this.</p></section> | |
| <section class="doc-section level-1"><h2 id="_wide_characters_and_utf_modes">Wide characters and UTF modes</h2><p>Code points less than 256 can be specified in patterns by either braced or | |
| unbraced hexadecimal escape sequences (for example, <code class="regex rouge"><span class="pcre2-semantic">\x{b3}</span></code> or <code class="regex rouge"><span class="pcre2-semantic">\xb3</span></code>). Larger | |
| values have to use braced sequences. Unbraced octal code points up to <code class="regex rouge"><span class="pcre2-semantic">\777</span></code> are | |
| also recognized; larger ones can be coded using <code class="regex rouge"><span class="pcre2-semantic">\o{...}</span></code>.</p> | |
| <p>The escape sequence <code class="regex rouge"><span class="pcre2-semantic">\N{U+<em>hex digits</em>}</span></code> is recognized as another way of | |
| specifying a Unicode character by code point in a UTF mode. It is not allowed | |
| in non-UTF mode.</p> | |
| <p>In UTF mode, repeat quantifiers apply to complete UTF characters, not to | |
| individual code units.</p> | |
| <p>In UTF mode, the dot metacharacter matches one UTF character instead of a | |
| single code unit.</p> | |
| <p>In UTF mode, capture group names are not restricted to ASCII, and may contain | |
| any Unicode letters and decimal digits, as well as underscore.</p> | |
| <p>The escape sequence <code class="regex rouge"><span class="pcre2-semantic">\C</span></code> can be used to match a single code unit in UTF mode, | |
| but its use can lead to some strange effects because it breaks up multi-unit | |
| characters (see the description of <code class="regex rouge"><span class="pcre2-semantic">\C</span></code> in the | |
| <a href="./pcre2pattern.html"><code>pcre2pattern</code></a> | |
| documentation). For this reason, there is a build-time option that disables | |
| support for <code class="regex rouge"><span class="pcre2-semantic">\C</span></code> completely. There is also a less draconian compile-time option | |
| for locking out the use of <code class="regex rouge"><span class="pcre2-semantic">\C</span></code> when a pattern is compiled.</p> | |
| <p>The use of <code class="regex rouge"><span class="pcre2-semantic">\C</span></code> is not supported by the alternative matching function | |
| <code class="function pcre2-api rouge"><span class="nf">pcre2_dfa_match()</span></code> when in UTF-8 or UTF-16 mode, that is, when a character | |
| may consist of more than one code unit. The use of <code class="regex rouge"><span class="pcre2-semantic">\C</span></code> in these modes provokes | |
| a match-time error. Also, the JIT optimization does not support <code class="regex rouge"><span class="pcre2-semantic">\C</span></code> in these | |
| modes. If JIT optimization is requested for a UTF-8 or UTF-16 pattern that | |
| contains <code class="regex rouge"><span class="pcre2-semantic">\C</span></code>, it will not succeed, and so when <code class="function pcre2-api rouge"><span class="nf">pcre2_match()</span></code> is called, | |
| the matching will be carried out by the interpretive function.</p> | |
| <p>The character escapes <code class="regex rouge"><span class="pcre2-semantic">\b</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\B</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\d</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\D</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\s</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\S</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\w</span></code>, and <code class="regex rouge"><span class="pcre2-semantic">\W</span></code> correctly test | |
| characters of any code value, but, by default, the characters that PCRE2 | |
| recognizes as digits, spaces, or word characters remain the same set as in | |
| non-UTF mode, all with code points less than 256. This remains true even when | |
| PCRE2 is built to include Unicode support, because to do otherwise would slow | |
| down matching in many common cases. Note that this also applies to <code class="regex rouge"><span class="pcre2-semantic">\b</span></code> | |
| and <code class="regex rouge"><span class="pcre2-semantic">\B</span></code>, because they are defined in terms of <code class="regex rouge"><span class="pcre2-semantic">\w</span></code> and <code class="regex rouge"><span class="pcre2-semantic">\W</span></code>. If you want | |
| to test for a wider sense of, say, "digit", you can use explicit Unicode | |
| property tests such as <code class="regex rouge"><span class="pcre2-semantic">\p{Nd}</span></code>. Alternatively, if you set the <code class="constant pcre2-api rouge"><span class="no">PCRE2_UCP</span></code> option, | |
| the way that the character escapes work is changed so that Unicode properties | |
| are used to determine which characters match, though there are some options | |
| that suppress this for individual escapes. For details see the section | |
| <a href="./pcre2pattern.html#generic-character-types">Generic character types</a> | |
| in the | |
| <a href="./pcre2pattern.html"><code>pcre2pattern</code></a> | |
| documentation.</p> | |
| <p>Like the escapes, characters that match the POSIX named character classes are | |
| all low-valued characters unless the <code class="constant pcre2-api rouge"><span class="no">PCRE2_UCP</span></code> option is set, but there is an | |
| option to override this.</p> | |
| <p>In contrast to the character escapes and character classes, the special | |
| horizontal and vertical white space escapes (<code class="regex rouge"><span class="pcre2-semantic">\h</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\H</span></code>, <code class="regex rouge"><span class="pcre2-semantic">\v</span></code>, and <code class="regex rouge"><span class="pcre2-semantic">\V</span></code>) do match | |
| all the appropriate Unicode characters, whether or not <code class="constant pcre2-api rouge"><span class="no">PCRE2_UCP</span></code> is set.</p></section> | |
| <section class="doc-section level-1"><h2 id="_unicode_case_equivalence">Unicode case-equivalence</h2><p>If either <code class="constant pcre2-api rouge"><span class="no">PCRE2_UTF</span></code> or <code class="constant pcre2-api rouge"><span class="no">PCRE2_UCP</span></code> is set, upper/lower case processing makes use | |
| of Unicode properties except for characters whose code points are less than 128 | |
| and that have at most two case-equivalent values. For these, a direct table | |
| lookup is used for speed. A few Unicode characters such as Greek sigma have | |
| more than two code points that are case-equivalent, and these are treated | |
| specially. Setting <code class="constant pcre2-api rouge"><span class="no">PCRE2_UCP</span></code> without <code class="constant pcre2-api rouge"><span class="no">PCRE2_UTF</span></code> allows Unicode-style case | |
| processing for non-UTF character encodings such as UCS-2.</p> | |
| <p>There are two ASCII characters (S and K) that, in addition to their ASCII lower | |
| case equivalents, have a non-ASCII one as well (long S and Kelvin sign). | |
| Recognition of these non-ASCII characters as case-equivalent to their ASCII | |
| counterparts can be disabled by setting the <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_CASELESS_RESTRICT</span></code> | |
| option. When this is set, all characters in a case equivalence must either be | |
| ASCII or non-ASCII; there can be no mixing.</p> | |
| <dl class="compact" style="--tab-width: 4"><div class="dlist-item"><dt><span>Without <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_CASELESS_RESTRICT</span></code></span></dt><dd>'k' = 'K' = <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-1"><code class="character pcre2-api rouge"><span class="sc">U+212A</span></code><span class="character-card" id="pcre2-character-1" role="tooltip" aria-label="KELVIN SIGN (U+212A)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="K"></span><span class="character-name" data-character-name="KELVIN SIGN"></span><span class="character-codepoint" data-codepoint="U+212A"></span></span></span> (Kelvin sign)<br> | |
| 's' = 'S' = <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-2"><code class="character pcre2-api rouge"><span class="sc">U+017F</span></code><span class="character-card" id="pcre2-character-2" role="tooltip" aria-label="LATIN SMALL LETTER LONG S (U+017F)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="ſ"></span><span class="character-name" data-character-name="LATIN SMALL LETTER LONG S"></span><span class="character-codepoint" data-codepoint="U+017F"></span></span></span> (long S)</dd></div><div class="dlist-item"><dt><span>With <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_CASELESS_RESTRICT</span></code></span></dt><dd>'k' = 'K'<br> | |
| <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-3"><code class="character pcre2-api rouge"><span class="sc">U+212A</span></code><span class="character-card" id="pcre2-character-3" role="tooltip" aria-label="KELVIN SIGN (U+212A)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="K"></span><span class="character-name" data-character-name="KELVIN SIGN"></span><span class="character-codepoint" data-codepoint="U+212A"></span></span></span> (Kelvin sign) — only case-equivalent to itself<br> | |
| 's' = 'S'<br> | |
| <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-4"><code class="character pcre2-api rouge"><span class="sc">U+017F</span></code><span class="character-card" id="pcre2-character-4" role="tooltip" aria-label="LATIN SMALL LETTER LONG S (U+017F)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="ſ"></span><span class="character-name" data-character-name="LATIN SMALL LETTER LONG S"></span><span class="character-codepoint" data-codepoint="U+017F"></span></span></span> (long S) — only case-equivalent to itself</dd></div></dl> | |
| <p>One language family, Turkish and Azeri, has its own case-insensitivity rules, | |
| which can be selected by setting <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_TURKISH_CASING</span></code>. This alters the | |
| behaviour of the 'i', 'I', <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-5"><code class="character pcre2-api rouge"><span class="sc">U+0130</span></code><span class="character-card" id="pcre2-character-5" role="tooltip" aria-label="LATIN CAPITAL LETTER I WITH DOT ABOVE (U+0130)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="İ"></span><span class="character-name" data-character-name="LATIN CAPITAL LETTER I WITH DOT ABOVE"></span><span class="character-codepoint" data-codepoint="U+0130"></span></span></span> (capital I with dot above), and <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-6"><code class="character pcre2-api rouge"><span class="sc">U+0131</span></code><span class="character-card" id="pcre2-character-6" role="tooltip" aria-label="LATIN SMALL LETTER DOTLESS I (U+0131)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="ı"></span><span class="character-name" data-character-name="LATIN SMALL LETTER DOTLESS I"></span><span class="character-codepoint" data-codepoint="U+0131"></span></span></span> | |
| (small dotless i) characters.</p> | |
| <dl class="compact" style="--tab-width: 4"><div class="dlist-item"><dt><span>Without <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_TURKISH_CASING</span></code></span></dt><dd>'i' = 'I'<br> | |
| <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-7"><code class="character pcre2-api rouge"><span class="sc">U+0130</span></code><span class="character-card" id="pcre2-character-7" role="tooltip" aria-label="LATIN CAPITAL LETTER I WITH DOT ABOVE (U+0130)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="İ"></span><span class="character-name" data-character-name="LATIN CAPITAL LETTER I WITH DOT ABOVE"></span><span class="character-codepoint" data-codepoint="U+0130"></span></span></span> (capital I with dot above) — only case-equivalent to itself<br> | |
| <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-8"><code class="character pcre2-api rouge"><span class="sc">U+0131</span></code><span class="character-card" id="pcre2-character-8" role="tooltip" aria-label="LATIN SMALL LETTER DOTLESS I (U+0131)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="ı"></span><span class="character-name" data-character-name="LATIN SMALL LETTER DOTLESS I"></span><span class="character-codepoint" data-codepoint="U+0131"></span></span></span> (small dotless i) — only case-equivalent to itself</dd></div><div class="dlist-item"><dt><span>With <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_TURKISH_CASING</span></code></span></dt><dd>'i' = <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-9"><code class="character pcre2-api rouge"><span class="sc">U+0130</span></code><span class="character-card" id="pcre2-character-9" role="tooltip" aria-label="LATIN CAPITAL LETTER I WITH DOT ABOVE (U+0130)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="İ"></span><span class="character-name" data-character-name="LATIN CAPITAL LETTER I WITH DOT ABOVE"></span><span class="character-codepoint" data-codepoint="U+0130"></span></span></span> (capital I with dot above)<br> | |
| <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-10"><code class="character pcre2-api rouge"><span class="sc">U+0131</span></code><span class="character-card" id="pcre2-character-10" role="tooltip" aria-label="LATIN SMALL LETTER DOTLESS I (U+0131)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="ı"></span><span class="character-name" data-character-name="LATIN SMALL LETTER DOTLESS I"></span><span class="character-codepoint" data-codepoint="U+0131"></span></span></span> (small dotless i) = 'I'</dd></div></dl> | |
| <p>It is not allowed to specify both <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_CASELESS_RESTRICT</span></code> and | |
| <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_TURKISH_CASING</span></code> together.</p> | |
| <p>From release 10.45 the Unicode letter properties <code class="regex rouge"><span class="pcre2-semantic">Lu</span></code> (upper case), <code class="regex rouge"><span class="pcre2-semantic">Ll</span></code> (lower | |
| case), and <code class="regex rouge"><span class="pcre2-semantic">Lt</span></code> (title case) are all treated as <code class="regex rouge"><span class="pcre2-semantic">Lc</span></code> (cased letter) when caseless | |
| matching is set by the <code class="constant pcre2-api rouge"><span class="no">PCRE2_CASELESS</span></code> option or <code class="regex rouge"><span class="pcre2-semantic">(?i)</span></code> within the pattern.</p></section> | |
| <section class="doc-section level-1"><h2 id="script-runs">Script runs</h2><p>The pattern constructs <code class="regex rouge"><span class="pcre2-semantic">(*script_run:...)</span></code> and <code class="regex rouge"><span class="pcre2-semantic">(*atomic_script_run:...)</span></code>, with | |
| synonyms <code class="regex rouge"><span class="pcre2-semantic">(*sr:...)</span></code> and <code class="regex rouge"><span class="pcre2-semantic">(*asr:...)</span></code>, verify that the string matched within the | |
| parentheses is a script run. In concept, a script run is a sequence of | |
| characters that are all from the same Unicode script. However, because some | |
| scripts are commonly used together, and because some diacritical and other | |
| marks are used with multiple scripts, it is not that simple.</p> | |
| <p>Every Unicode character has a Script property, mostly with a value | |
| corresponding to the name of a script, such as Latin, Greek, or Cyrillic. There | |
| are also three special values:</p> | |
| <p>"Unknown" is used for code points that have not been assigned, and also for the | |
| surrogate code points. In the PCRE2 32-bit library, characters whose code | |
| points are greater than the Unicode maximum (<span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-11"><code class="character pcre2-api rouge"><span class="sc">U+10FFFF</span></code><span class="character-card" id="pcre2-character-11" role="tooltip" aria-label="<noncharacter-10FFFF> (U+10FFFF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="<noncharacter-10FFFF>"></span><span class="character-codepoint" data-codepoint="U+10FFFF"></span></span></span>), which are accessible | |
| only in non-UTF mode, are assigned the Unknown script.</p> | |
| <p>"Common" is used for characters that are used with many scripts. These include | |
| punctuation, emoji, mathematical, musical, and currency symbols, and the ASCII | |
| digits 0 to 9.</p> | |
| <p>"Inherited" is used for characters such as diacritical marks that modify a | |
| previous character. These are considered to take on the script of the character | |
| that they modify.</p> | |
| <p>Some Inherited characters are used with many scripts, but many of them are only | |
| normally used with a small number of scripts. For example, <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-12"><code class="character pcre2-api rouge"><span class="sc">U+102E0</span></code><span class="character-card" id="pcre2-character-12" role="tooltip" aria-label="COPTIC EPACT THOUSANDS MARK (U+102E0)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="◌𐋠"></span><span class="character-name" data-character-name="COPTIC EPACT THOUSANDS MARK"></span><span class="character-codepoint" data-codepoint="U+102E0"></span></span></span> (Coptic | |
| Epact thousands mark) is used only with Arabic and Coptic. In order to make it | |
| possible to check this, a Unicode property called Script Extension exists. Its | |
| value is a list of scripts that apply to the character. For the majority of | |
| characters, the list contains just one script, the same one as the Script | |
| property. However, for characters such as <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-13"><code class="character pcre2-api rouge"><span class="sc">U+102E0</span></code><span class="character-card" id="pcre2-character-13" role="tooltip" aria-label="COPTIC EPACT THOUSANDS MARK (U+102E0)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="◌𐋠"></span><span class="character-name" data-character-name="COPTIC EPACT THOUSANDS MARK"></span><span class="character-codepoint" data-codepoint="U+102E0"></span></span></span> more than one Script is | |
| listed. There are also some Common characters that have a single, non-Common | |
| script in their Script Extension list.</p> | |
| <p>The section <a href="#basic-script-run-rules">Basic script run rules</a> describes the basic rules for deciding whether a given string | |
| of characters is a script run. Note, however, that there are some special cases | |
| involving the Chinese Han script, and an additional constraint for decimal | |
| digits. These are covered in subsequent sections.</p> | |
| <section class="doc-section level-2"><h3 id="basic-script-run-rules">Basic script run rules</h3><p>A string that is less than two characters long is a script run. This is the | |
| only case in which an Unknown character can be part of a script run. Longer | |
| strings are checked using only the Script Extensions property, not the basic | |
| Script property.</p> | |
| <p>If a character’s Script Extension property is the single value "Inherited", it | |
| is always accepted as part of a script run. This is also true for the property | |
| "Common", subject to the checking of decimal digits described below. All the | |
| remaining characters in a script run must have at least one script in common in | |
| their Script Extension lists. In set-theoretic terminology, the intersection of | |
| all the sets of scripts must not be empty.</p> | |
| <p>A simple example is an Internet name such as <code class="subject rouge"><span class="pcre2-semantic">google.com</span></code>. The letters are all | |
| in the Latin script, and the dot is Common, so this string is a script run. | |
| However, the Cyrillic letter <code class="subject rouge"><span class="pcre2-semantic">o</span></code> looks exactly the same as the Latin <code class="subject rouge"><span class="pcre2-semantic">o</span></code>; a | |
| string that looks the same, but with Cyrillic <code class="subject rouge"><span class="pcre2-semantic">o</span></code>s is not a script run.</p> | |
| <p>More interesting examples involve characters with more than one script in their | |
| Script Extension. Consider the following characters:</p> | |
| <ul class="compact"><li><span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-14"><code class="character pcre2-api rouge"><span class="sc">U+060C</span></code><span class="character-card" id="pcre2-character-14" role="tooltip" aria-label="ARABIC COMMA (U+060C)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="،"></span><span class="character-name" data-character-name="ARABIC COMMA"></span><span class="character-codepoint" data-codepoint="U+060C"></span></span></span>, Arabic comma</li><li><span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-15"><code class="character pcre2-api rouge"><span class="sc">U+06D4</span></code><span class="character-card" id="pcre2-character-15" role="tooltip" aria-label="ARABIC FULL STOP (U+06D4)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="۔"></span><span class="character-name" data-character-name="ARABIC FULL STOP"></span><span class="character-codepoint" data-codepoint="U+06D4"></span></span></span>, Arabic full stop</li></ul> | |
| <p>The first has the Script Extension list Arabic, Hanifi Rohingya, Syriac, and | |
| Thaana; the second has just Arabic and Hanifi Rohingya. Both of them could | |
| appear in script runs of either Arabic or Hanifi Rohingya. The first could also | |
| appear in Syriac or Thaana script runs, but the second could not.</p></section> | |
| <section class="doc-section level-2"><h3 id="_the_chinese_han_script">The Chinese Han script</h3><p>The Chinese Han script is commonly used in conjunction with other scripts for | |
| writing certain languages. Japanese uses the Hiragana and Katakana scripts | |
| together with Han; Korean uses Hangul and Han; Taiwanese Mandarin uses Bopomofo | |
| and Han. These three combinations are treated as special cases when checking | |
| script runs and are, in effect, "virtual scripts". Thus, a script run may | |
| contain a mixture of Hiragana, Katakana, and Han, or a mixture of Hangul and | |
| Han, or a mixture of Bopomofo and Han, but not, for example, a mixture of | |
| Hangul and Bopomofo and Han. PCRE2 (like Perl) follows Unicode’s Technical | |
| Standard 39 ("Unicode Security Mechanisms", <a class="bare" href="http://unicode.org/reports/tr39/">http://unicode.org/reports/tr39/</a>) | |
| in allowing such mixtures.</p></section> | |
| <section class="doc-section level-2"><h3 id="_decimal_digits">Decimal digits</h3><p>Unicode contains many sets of 10 decimal digits in different scripts, and some | |
| scripts (including the Common script) contain more than one set. Some of these | |
| decimal digits them are visually indistinguishable from the common ASCII | |
| digits. In addition to the script checking described above, if a script run | |
| contains any decimal digits, they must all come from the same set of 10 | |
| adjacent characters.</p></section></section> | |
| <section class="doc-section level-1"><h2 id="_validity_of_utf_strings">Validity of UTF strings</h2><p>When the <code class="constant pcre2-api rouge"><span class="no">PCRE2_UTF</span></code> option is set, the strings passed as patterns and subjects | |
| are (by default) checked for validity on entry to the relevant functions. If an | |
| invalid UTF string is passed, a negative error code is returned. The code unit | |
| offset to the offending character can be extracted from the match data block by | |
| calling <code class="function pcre2-api rouge"><span class="nf">pcre2_get_startchar()</span></code>, which is used for this purpose after a UTF | |
| error.</p> | |
| <p>In some situations, you may already know that your strings are valid, and | |
| therefore want to skip these checks in order to improve performance, for | |
| example in the case of a long subject string that is being scanned repeatedly. | |
| If you set the <code class="constant pcre2-api rouge"><span class="no">PCRE2_NO_UTF_CHECK</span></code> option at compile time or at match time, | |
| PCRE2 assumes that the pattern or subject it is given (respectively) contains | |
| only valid UTF code unit sequences.</p> | |
| <p>If you pass an invalid UTF string when <code class="constant pcre2-api rouge"><span class="no">PCRE2_NO_UTF_CHECK</span></code> is set, the result | |
| is undefined and your program may crash or loop indefinitely or give incorrect | |
| results. There is, however, one mode of matching that can handle invalid UTF | |
| subject strings. This is enabled by passing <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> to | |
| <code class="function pcre2-api rouge"><span class="nf">pcre2_compile()</span></code> and is discussed below in the section | |
| <a href="#utf8-string-errors">Errors in UTF-8 strings</a>. The rest of | |
| this section covers the case when <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> is not set.</p> | |
| <p>Passing <code class="constant pcre2-api rouge"><span class="no">PCRE2_NO_UTF_CHECK</span></code> to <code class="function pcre2-api rouge"><span class="nf">pcre2_compile()</span></code> just disables the UTF check | |
| for the pattern; it does not also apply to subject strings. If you want to | |
| disable the check for a subject string you must pass this same option to | |
| <code class="function pcre2-api rouge"><span class="nf">pcre2_match()</span></code> or <code class="function pcre2-api rouge"><span class="nf">pcre2_dfa_match()</span></code>.</p> | |
| <p>UTF-16 and UTF-32 strings can indicate their endianness by special code knows | |
| as a byte-order mark (BOM). The PCRE2 functions do not handle this, expecting | |
| strings to be in host byte order.</p> | |
| <p>Unless <code class="constant pcre2-api rouge"><span class="no">PCRE2_NO_UTF_CHECK</span></code> is set, a UTF string is checked before any other | |
| processing takes place. In the case of <code class="function pcre2-api rouge"><span class="nf">pcre2_match()</span></code> and | |
| <code class="function pcre2-api rouge"><span class="nf">pcre2_dfa_match()</span></code> calls with a non-zero starting offset, the check is | |
| applied only to that part of the subject that could be inspected during | |
| matching, and there is a check that the starting offset points to the first | |
| code unit of a character or to the end of the subject. If there are no | |
| lookbehind assertions in the pattern, the check starts at the starting offset. | |
| Otherwise, it starts at the length of the longest lookbehind before the | |
| starting offset, or at the start of the subject if there are not that many | |
| characters before the starting offset. Note that the sequences <code class="regex rouge"><span class="pcre2-semantic">\b</span></code> and <code class="regex rouge"><span class="pcre2-semantic">\B</span></code> are | |
| one-character lookbehinds.</p> | |
| <p>In addition to checking the format of the string, there is a check to ensure | |
| that all code points lie in the range <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-16"><code class="character pcre2-api rouge"><span class="sc">U+0</span></code><span class="character-card" id="pcre2-character-16" role="tooltip" aria-label="NULL (U+0000)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="NULL"></span><span class="character-codepoint" data-codepoint="U+0000"></span></span></span> to <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-17"><code class="character pcre2-api rouge"><span class="sc">U+10FFFF</span></code><span class="character-card" id="pcre2-character-17" role="tooltip" aria-label="<noncharacter-10FFFF> (U+10FFFF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="<noncharacter-10FFFF>"></span><span class="character-codepoint" data-codepoint="U+10FFFF"></span></span></span>, excluding the surrogate | |
| area. The so-called "non-character" code points are not excluded because | |
| Unicode corrigendum #9 makes it clear that they should not be.</p> | |
| <p>Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16, | |
| where they are used in pairs to encode code points with values greater than | |
| <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-18"><code class="character pcre2-api rouge"><span class="sc">0xFFFF</span></code><span class="character-card" id="pcre2-character-18" role="tooltip" aria-label="<noncharacter-FFFF> (U+FFFF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="<noncharacter-FFFF>"></span><span class="character-codepoint" data-codepoint="U+FFFF"></span></span></span>. The code points that are encoded by UTF-16 pairs are available | |
| independently in the UTF-8 and UTF-32 encodings. (In other words, the whole | |
| surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and | |
| UTF-32.)</p> | |
| <p>Setting <code class="constant pcre2-api rouge"><span class="no">PCRE2_NO_UTF_CHECK</span></code> at compile time does not disable the error that is | |
| given if an escape sequence for an invalid Unicode code point is encountered in | |
| the pattern. If you want to allow escape sequences such as <code class="regex rouge"><span class="pcre2-semantic">\x{d800}</span></code> (a | |
| surrogate code point) you can set the <code class="constant pcre2-api rouge"><span class="no">PCRE2_EXTRA_ALLOW_SURROGATE_ESCAPES</span></code> extra | |
| option. However, this is possible only in UTF-8 and UTF-32 modes, because these | |
| values are not representable in UTF-16.</p> | |
| <section class="doc-section level-2"><h3 id="utf8-string-errors">Errors in UTF-8 strings</h3><p>The following negative error codes are given for invalid UTF-8 strings:</p> | |
| <dl class="errors spaced" style="--tab-width: 12"><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR1</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR2</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR3</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR4</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR5</span></code></span></dt><dd>The string ends with a truncated UTF-8 character; the code specifies how many | |
| bytes are missing (1 to 5). Although RFC 3629 restricts UTF-8 characters to be | |
| no longer than 4 bytes, the encoding scheme (originally defined by RFC 2279) | |
| allows for up to 6 bytes, and this is checked first; hence the possibility of | |
| 4 or 5 missing bytes.</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR6</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR7</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR8</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR9</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR10</span></code></span></dt><dd>The two most significant bits of the 2nd, 3rd, 4th, 5th, or 6th byte of the | |
| character do not have the binary value <code class="c rouge" data-lang="c"><span class="mi">0</span><span class="nv">b10</span></code> (that is, either the most | |
| significant bit is 0, or the next bit is 1).</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR11</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR12</span></code></span></dt><dd>A character that is valid by the RFC 2279 rules is either 5 or 6 bytes long; | |
| these code points are excluded by RFC 3629.</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR13</span></code></span></dt><dd>A 4-byte character has a value greater than <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-19"><code class="character pcre2-api rouge"><span class="sc">0x10ffff</span></code><span class="character-card" id="pcre2-character-19" role="tooltip" aria-label="<noncharacter-10FFFF> (U+10FFFF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="<noncharacter-10FFFF>"></span><span class="character-codepoint" data-codepoint="U+10FFFF"></span></span></span>; these code points are | |
| excluded by RFC 3629.</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR14</span></code></span></dt><dd>A 3-byte character has a value in the range <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-20"><code class="character pcre2-api rouge"><span class="sc">0xd800</span></code><span class="character-card" id="pcre2-character-20" role="tooltip" aria-label="Surrogate code point (U+D800)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="Surrogate code point"></span><span class="character-codepoint" data-codepoint="U+D800"></span></span></span> to <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-21"><code class="character pcre2-api rouge"><span class="sc">0xdfff</span></code><span class="character-card" id="pcre2-character-21" role="tooltip" aria-label="Surrogate code point (U+DFFF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="Surrogate code point"></span><span class="character-codepoint" data-codepoint="U+DFFF"></span></span></span>; this range of | |
| code points are reserved by RFC 3629 for use with UTF-16, and so are excluded | |
| from UTF-8.</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR15</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR16</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR17</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR18</span></code><br><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR19</span></code></span></dt><dd>A 2-, 3-, 4-, 5-, or 6-byte character is "overlong", that is, it codes for a | |
| value that can be represented by fewer bytes, which is invalid. For example, | |
| the two bytes <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-22"><code class="character pcre2-api rouge"><span class="sc">0xc0</span></code><span class="character-card" id="pcre2-character-22" role="tooltip" aria-label="LATIN CAPITAL LETTER A WITH GRAVE (U+00C0)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="À"></span><span class="character-name" data-character-name="LATIN CAPITAL LETTER A WITH GRAVE"></span><span class="character-codepoint" data-codepoint="U+00C0"></span></span></span>, <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-23"><code class="character pcre2-api rouge"><span class="sc">0xae</span></code><span class="character-card" id="pcre2-character-23" role="tooltip" aria-label="REGISTERED SIGN (U+00AE)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="®"></span><span class="character-name" data-character-name="REGISTERED SIGN"></span><span class="character-codepoint" data-codepoint="U+00AE"></span></span></span> give the value <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-24"><code class="character pcre2-api rouge"><span class="sc">0x2e</span></code><span class="character-card" id="pcre2-character-24" role="tooltip" aria-label="FULL STOP (U+002E)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="."></span><span class="character-name" data-character-name="FULL STOP"></span><span class="character-codepoint" data-codepoint="U+002E"></span></span></span>, whose correct coding uses just | |
| one byte.</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR20</span></code></span></dt><dd>The two most significant bits of the first byte of a character have the binary | |
| value <code class="c rouge" data-lang="c"><span class="mi">0</span><span class="nv">b10</span></code> (that is, the most significant bit is 1 and the second is 0). Such a | |
| byte can only validly occur as the second or subsequent byte of a multi-byte | |
| character.</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF8_ERR21</span></code></span></dt><dd>The first byte of a character has the value <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-25"><code class="character pcre2-api rouge"><span class="sc">0xfe</span></code><span class="character-card" id="pcre2-character-25" role="tooltip" aria-label="LATIN SMALL LETTER THORN (U+00FE)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="þ"></span><span class="character-name" data-character-name="LATIN SMALL LETTER THORN"></span><span class="character-codepoint" data-codepoint="U+00FE"></span></span></span> or <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-26"><code class="character pcre2-api rouge"><span class="sc">0xff</span></code><span class="character-card" id="pcre2-character-26" role="tooltip" aria-label="LATIN SMALL LETTER Y WITH DIAERESIS (U+00FF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character="ÿ"></span><span class="character-name" data-character-name="LATIN SMALL LETTER Y WITH DIAERESIS"></span><span class="character-codepoint" data-codepoint="U+00FF"></span></span></span>. These values can | |
| never occur in a valid UTF-8 string.</dd></div></dl></section> | |
| <section class="doc-section level-2"><h3 id="_errors_in_utf_16_strings">Errors in UTF-16 strings</h3><p>The following negative error codes are given for invalid UTF-16 strings:</p> | |
| <dl class="errors compact" style="--tab-width: 12"><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF16_ERR1</span></code></span></dt><dd>Missing low surrogate at end of string</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF16_ERR2</span></code></span></dt><dd>Invalid low surrogate follows high surrogate</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF16_ERR3</span></code></span></dt><dd>Isolated low surrogate</dd></div></dl></section> | |
| <section class="doc-section level-2"><h3 id="_errors_in_utf_32_strings">Errors in UTF-32 strings</h3><p>The following negative error codes are given for invalid UTF-32 strings:</p> | |
| <dl class="errors compact" style="--tab-width: 12"><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF32_ERR1</span></code></span></dt><dd>Surrogate character (<span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-27"><code class="character pcre2-api rouge"><span class="sc">0xd800</span></code><span class="character-card" id="pcre2-character-27" role="tooltip" aria-label="Surrogate code point (U+D800)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="Surrogate code point"></span><span class="character-codepoint" data-codepoint="U+D800"></span></span></span> to <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-28"><code class="character pcre2-api rouge"><span class="sc">0xdfff</span></code><span class="character-card" id="pcre2-character-28" role="tooltip" aria-label="Surrogate code point (U+DFFF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="Surrogate code point"></span><span class="character-codepoint" data-codepoint="U+DFFF"></span></span></span>)</dd></div><div class="dlist-item"><dt><span><code class="constant pcre2-api rouge"><span class="no">PCRE2_ERROR_UTF32_ERR2</span></code></span></dt><dd>Code point is greater than <span class="character-info" title="" tabindex="0" aria-describedby="pcre2-character-29"><code class="character pcre2-api rouge"><span class="sc">0x10ffff</span></code><span class="character-card" id="pcre2-character-29" role="tooltip" aria-label="<noncharacter-10FFFF> (U+10FFFF)" aria-hidden="true" data-pagefind-ignore=""><span class="character-glyph" data-character=""></span><span class="character-name" data-character-name="<noncharacter-10FFFF>"></span><span class="character-codepoint" data-codepoint="U+10FFFF"></span></span></span></dd></div></dl></section></section> | |
| <section class="doc-section level-1"><h2 id="_matching_in_invalid_utf_strings">Matching in invalid UTF strings</h2><p>You can run pattern matches on subject strings that may contain invalid UTF | |
| sequences if you call <code class="function pcre2-api rouge"><span class="nf">pcre2_compile()</span></code> with the <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> | |
| option. This is supported by <code class="function pcre2-api rouge"><span class="nf">pcre2_match()</span></code>, including JIT matching, but | |
| not by <code class="function pcre2-api rouge"><span class="nf">pcre2_dfa_match()</span></code>. When <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> is set, it forces | |
| <code class="constant pcre2-api rouge"><span class="no">PCRE2_UTF</span></code> to be set as well. Note, however, that the pattern itself must be a | |
| valid UTF string.</p> | |
| <p>If you do not set <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> when calling <code class="function pcre2-api rouge"><span class="nf">pcre2_compile()</span></code>, and | |
| you are not certain that your subject strings are valid UTF sequences, you | |
| should not make use of the JIT "fast path" function <code class="function pcre2-api rouge"><span class="nf">pcre2_jit_match()</span></code> | |
| because it bypasses sanity checks, including the one for UTF validity. An | |
| invalid string may cause undefined behaviour, including looping, crashing, or | |
| giving the wrong answer.</p> | |
| <p>Setting <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> does not affect what <code class="function pcre2-api rouge"><span class="nf">pcre2_compile()</span></code> | |
| generates, but if <code class="function pcre2-api rouge"><span class="nf">pcre2_jit_compile()</span></code> is subsequently called, it does | |
| generate different code. If JIT is not used, the option affects the behaviour | |
| of the interpretive code in <code class="function pcre2-api rouge"><span class="nf">pcre2_match()</span></code>. When <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code> | |
| is set at compile time, <code class="constant pcre2-api rouge"><span class="no">PCRE2_NO_UTF_CHECK</span></code> is ignored at match time.</p> | |
| <p>In this mode, an invalid code unit sequence in the subject never matches any | |
| pattern item. It does not match dot, it does not match <code class="regex rouge"><span class="pcre2-semantic">\p{Any}</span></code>, it does not | |
| even match negative items such as <code class="regex rouge"><span class="pcre2-semantic">[^X]</span></code>. A lookbehind assertion fails if it | |
| encounters an invalid sequence while moving the current point backwards. In | |
| other words, an invalid UTF code unit sequence acts as a barrier which no match | |
| can cross.</p> | |
| <p>You can also think of this as the subject being split up into fragments of | |
| valid UTF, delimited internally by invalid code unit sequences. The pattern is | |
| matched fragment by fragment. The result of a successful match, however, is | |
| given as code unit offsets in the entire subject string in the usual way. There | |
| are a few points to consider:</p> | |
| <p>The internal boundaries are not interpreted as the beginnings or ends of lines | |
| and so do not match circumflex or dollar characters in the pattern.</p> | |
| <p>If <code class="function pcre2-api rouge"><span class="nf">pcre2_match()</span></code> is called with an offset that points to an invalid | |
| UTF-sequence, that sequence is skipped, and the match starts at the next valid | |
| UTF character, or the end of the subject.</p> | |
| <p>At internal fragment boundaries, <code class="regex rouge"><span class="pcre2-semantic">\b</span></code> and <code class="regex rouge"><span class="pcre2-semantic">\B</span></code> behave in the same way as at the | |
| beginning and end of the subject. For example, a sequence such as <code class="regex rouge"><span class="pcre2-semantic">\bWORD\b</span></code> | |
| would match an instance of <code class="subject rouge"><span class="pcre2-semantic">WORD</span></code> that is surrounded by invalid UTF code units.</p> | |
| <p>Using <code class="constant pcre2-api rouge"><span class="no">PCRE2_MATCH_INVALID_UTF</span></code>, an application can run matches on arbitrary | |
| data, knowing that any matched strings that are returned are valid UTF. This | |
| can be useful when searching for UTF text in executable or other binary files.</p> | |
| <p>Note, however, that the 16-bit and 32-bit PCRE2 libraries process strings as | |
| sequences of <code class="type pcre2-api rouge"><span class="kt">uint16_t</span></code> or <code class="type pcre2-api rouge"><span class="kt">uint32_t</span></code> code points. They cannot find valid UTF | |
| sequences within an arbitrary string of bytes unless such sequences are | |
| suitably aligned.</p> | |
| <h2 id="_author" class="discrete">Author</h2> | |
| <p>Philip Hazel<br> | |
| Retired from University Computing Service<br> | |
| Cambridge, England.</p> | |
| <h2 id="_revision" class="discrete">Revision</h2> | |
| <p>Last updated: 27 November 2024<br> | |
| Copyright © 1997-2024 University of Cambridge.</p></section></article></div></main><footer class="offline-footer"></footer></body></html> |