ramaraihan12/tiny-triage

tiny-triage

A small classifier that reads a GitHub issue and says whether it is a bug, a feature request, a question or a docs problem, and whether the maintainers would probably ask for more information. It runs in your browser: the issue text never leaves this page.

What it is

An ELECTRA-small encoder (14M parameters, 14 MB after int8 quantization) fine-tuned on the labels maintainers of public repos put on their own issues. It was trained on repos, tuned on and tested on repos it never saw, so the numbers below say something about a new project, not just about projects it memorised.

category

macro-F1 on test issues, TF-IDF baseline , always guessing the most common class . The gap to the baseline is (95% CI ).

needs_info experimental

macro-F1 on test issues at the demo's threshold (), against for the TF-IDF baseline with its own tuned threshold. At the default 0.5 it is against . On repos it has not seen, this output is worse than the baseline (the gap is , 95% CI ), so treat it as a hint.

Macro-F1 averages the per-class F1 scores, so rare classes count as much as common ones. The model and baseline numbers are on the same text; the test set is six repos, so the confidence interval resamples issues only and is too tight for a new repo. This page runs the int8 model on text cleaned the way the demo does it, which scores category against for the baseline and needs_info against on the same test issues. Method and caveats are in the README.

Try it

Paste an issue, or paste the URL of a public GitHub issue and the page fetches it from api.github.com with no token (GitHub allows 60 such requests an hour per IP address).

50 test issues, three systems

A fixed random sample of 50 issues from the six test repos (stratified by repo and category, ), each compared with what the maintainers did. The model and the baseline saw the same cleaned text; Llama 3.3 got that text too, with the prompt from my issue-triage-action. Only titles and links are shown; the issue bodies are not published.

50 test issues with maintainer labels and predictions of three systems
Issue Maintainer TF-IDF tiny-triage Llama 3.3 Built with Llama

Each cell is category / needs_info. Green means it matches the maintainer, red means it does not. needs_info is only scored where the repo has such a label; httpx, flask and pydantic do not, so the maintainer column shows no needs_info for them. 50 issues is a small sample: a difference of a few issues between systems is noise.

Limits