Knowledge Graph Evaluation: Scoring 12 Real Decisions
Three weeks ago I started scoring my knowledge graph against the answer I would have reached without it. It earns its place: 6 of 12 decisions came out sharper, 2 errors never reached real work, and 73 minutes of research time saved.
The Metric That Cannot Fail
Retrieval told me none of that. 21 of 21 queries, 100% recall, 8s median. Recall and latency describe the index, not the work. No input makes them come back bad.
Freeze Your Answer First
None of that score exists unless the old answer is written down first: the options, the criteria, the recommendation, a confidence number. Then query. Write the baseline afterwards and you rebuild a past self who conveniently agreed with whatever came back.
If your retrieval has never contradicted you, nobody has checked.
A knowledge graph evaluation that has never once contradicted you has not been checked yet, it has been admired. The field notes cover how these decision measurements get built, and where they keep breaking.
Your agents answer from whatever the retriever finds, and too often that is last quarter's truth. I build the context layer they answer and act from: a temporal knowledge graph that keeps every fact with its source and the time it held, reads with each person's own permissions, and writes nothing without a person's approval. On your own tenant, billed by the hour, step by step.