EVIDENCE · QUALITY AND PERFORMANCE

Benchmarks

These measurements are evidence for fixed inputs and settings. A high Canonical score does not establish recall for typos, spacing errors, or new repositories. Compare both precision and recall for smart and any in the Robust results below, then check fixed queries on your own documents. See the source reports for the sample scope and limitations of real-document evaluations.

Evaluation scope

Canonical contains 500 positive and 500 negative sentences manually reviewed for standard Korean. kfind, Kiwi, Lindera, MeCab-ko, and KOMORAN perform the same lemma, POS, and sentence task.

Robust contains 250 positive and 250 negative natural sentences with real errors. Its results are not combined with Canonical, and the robustness setting, error location, and denominator are recorded separately.

The query matrix applies multiple positive and negative queries to each sentence. It measures partial recovery and within-sentence false positives. A versioned product-contract registry defines its reviewed expectations.

Raw and contract-adjusted metrics

Raw metrics preserve the source corpus gold. TP, FP, TN, FN, precision, recall, and F1 are reported without modification.

Contract-adjusted metrics apply a registry fixed before product execution to the same predictions. It may reclassify semantically indistinguishable homographs, source-aligned internal components, and gold-span errors. Only nonstandard input outside the product contract may be excluded. Unsupported or expensive grammar remains in the denominator.

A contract-adjusted confusion matrix adds a superscript c to each raw abbreviation: TPᶜ, FPᶜ, TNᶜ, and FNᶜ. A dataset without contract reviews keeps its raw expectations and records a reviewed-case count of zero.

Canonical quality and performance

Confusion matrix

Same-workload performance

Fresh processes run one warm-up followed by five measurements. Quality and cost remain separate.

Query-matrix quality and performance

The query matrix selects up to three lemma-POS-span queries that should match in one source sentence and pairs each with a same-POS query that should not match. These values are aggregated per query and remain separate from the Canonical regression baseline.

Confusion matrix

Same-workload performance

Fresh processes run one warm-up followed by five measurements. Quality and cost remain separate.

Robust quality and performance

Confusion matrix

Same-workload performance

Fresh processes run one warm-up followed by five measurements. Quality and cost remain separate.

Morphology queries and regex baselines

Seven full-POS morphology queries with explicit POS run once with any and once with smart. The results are compared with a regex that manually enumerates inflected surfaces and another that lists only short stems. Each query has eight positives and eight negatives, for 112 cases.

This constructed fixture diagnoses coverage and boundary trade-offs for the same queries. It is neither a held-out quality benchmark nor a general ranking of Korean search quality.

Contract-adjusted values apply expectations fixed before execution to each method’s same predictions. The false-positive difference between any and smart remains visible; adjusted results never replace raw errors.

Confusion matrix

Seven-query batch time

d28fa2790b5592710291c5a90220698065ff8999 · 40e3c1fcfc7b803fe48121d8e73e7e7dde2a61de70f90213890e58b43bec5021

Benchmark contract and reports

Source evidence

The approved snapshot records the source report’s Git revision and SHA-256. The report contains the environment, fixture checksums, tool versions, and case-level failures.

d28fa2790b5592710291c5a90220698065ff8999 · 3258d172ce7098f9434150fb24d9a59fbedc73ec4dd0b84b17ca24062a68af28

Benchmark contract and reports