Gold Standard for Testing and Quality Assurance

The Gold Standard is a framework for defining test queries and expected results to measure and ensure search quality. It supports continuous monitoring by allowing daily execution to assess the quality consistently.

• questions - Contains arrays of queries with expected outputs.
• q - A list of test queries.
• ids - a list of ids
  • 1044059 - the id of the expected document
  • top10 - Document must appear in the top 10 results.
  • top1, top3 ... topN - N > 0
• for every query all ids are applied in the computation of the scores
• tags - a list of tags
  • for each tag a score is computed
  • that allows to see results analyse the result for keyword or semantic searches
• comment - can contain any text, for example:
  • describe the expected behaviour or results
  • describe the purpose of the queries

{
  "questions": [
    {
      "q": [
        "5g device",
        "5 g device"
      ],
      "ids": [
        "1044059 | top10",
        "1043570 | top10",
        "889814 | top10"
      ],
      "tags": [
        "semantic",
        "keyword"
      ],
      "comment": "Test queries with the same meaning but with a whitespace."
    }
  ]
}

questions:
  - q:
      - "5g device"
      - "5 g device"
    ids:
      - "1044059 | top10"
      - "1043570 | top10"
      - "889814 | top10"
    tags:
      - semantic
      - keyword
    comment: "Test queries with the same meaning but with a whitespace."