Question about the evaluation of OpenBookQA

[xumingyu2021]

It seems that the fact is not provided to give a answer, as a openbook qa, it may be better to use the fact, otherwise it become "closebookqa"
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/openbookqa/openbookqa.yaml

task: openbookqa
dataset_path: openbookqa
dataset_name: main #here maybe additional
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: test
doc_to_text: question_stem #here may need use fact1
doc_to_target: "{{choices.label.index(answerKey.lstrip())}}"
doc_to_choice: "{{choices.text}}"
should_decontaminate: true
doc_to_decontamination_query: question_stem
metric_list:

• metric: acc
  aggregation: mean
  higher_is_better: true
• metric: acc_norm
  aggregation: mean
  higher_is_better: true
  metadata:
  version: 1.0