What it is
An ELECTRA-small encoder (14M parameters, 14 MB after int8 quantization) fine-tuned on the labels maintainers of public repos put on their own issues. It was trained on repos, tuned on and tested on repos it never saw, so the numbers below say something about a new project, not just about projects it memorised.
category
macro-F1 on test issues, TF-IDF baseline , always guessing the most common class . The gap to the baseline is (95% CI ).
needs_info experimental
macro-F1 on test issues at the demo's threshold (), against for the TF-IDF baseline with its own tuned threshold. At the default 0.5 it is against . On repos it has not seen, this output is worse than the baseline (the gap is , 95% CI ), so treat it as a hint.
Macro-F1 averages the per-class F1 scores, so rare classes count as much as common ones. The model and baseline numbers are on the same text; the test set is six repos, so the confidence interval resamples issues only and is too tight for a new repo. This page runs the int8 model on text cleaned the way the demo does it, which scores category against for the baseline and needs_info against on the same test issues. Method and caveats are in the README.
Try it
Paste an issue, or paste the URL of a public GitHub issue and the page fetches it from api.github.com with no token (GitHub allows 60 such requests an hour per IP address).
Category
needs_info experimental
On repos it has not seen, needs_info is worse than the TF-IDF baseline. It is a hint about whether a maintainer would ask for details, nothing more.
Text the model saw
50 test issues, three systems
A fixed random sample of 50 issues from the six test repos (stratified by repo and category, ), each compared with what the maintainers did. The model and the baseline saw the same cleaned text; Llama 3.3 got that text too, with the prompt from my issue-triage-action. Only titles and links are shown; the issue bodies are not published.
| Issue | Maintainer | TF-IDF | tiny-triage | Llama 3.3 Built with Llama |
|---|
Each cell is category / needs_info. Green means it matches the maintainer, red means it does not. needs_info is only scored where the repo has such a label; httpx, flask and pydantic do not, so the maintainer column shows no needs_info for them. 50 issues is a small sample: a difference of a few issues between systems is noise.
Limits
- Labels come from large, active, mostly English-language projects with issue forms. A small repo with other habits may behave differently.
questionis the weakest class; many of them look like bug reports.- The category numbers are on six repos. Per repo they range from about 0.4 to 0.7 macro-F1.
- The model was trained on text where each repo's own template lines had been removed. This page removes comments, long code, links and checklists but no template lines, so issues that carry a form template reach the model with the template headings still in. On the test repos that made no measurable difference.