We are very glad to announce the first ELLIS AMS team is going to showcase their results of the Beyond the Imitation Game Benchmark (BIG-bench) collaborative task. The talk will happen on Tuesday 5th October, 16:00. The seminar will be held online, the zoom meeting link will be sent out later.<\/p>\n
<\/p>\n
Abstract:<\/p>\n
The Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark intended to probe large language models and provide concrete evidence of their capabilities and limitations. In the community building spirit of ELLIS-Amsterdam, we have formed three teams mixing Bachelor\u2019s, Master\u2019s, and PhD students and have contributed three tasks to the benchmark. In the seminar, we will briefly introduce the BIG-bench challenge and then the three teams will present their benchmarking tasks.<\/p>\n
The Metaphor Understanding task tests the capability of language models to understand English metaphors. It consists of two subtasks: in the first one, a language model is asked to correctly map a metaphorical expression to its correct literal paraphrases; in the second one, the model needs to map a literal paraphrase to the corresponding metaphorical expression. The two subtasks form a new dataset that takes into account the lessons learned from existing models and benchmarks.<\/p>\n
The Implicit Relations task evaluates a model\u2019s ability to infer relations between characters from short passages of English narratives, where the relations are left implicit. In each example, a passage and a question of the form \u201cWhat is X to Y?\u201d is presented, and the model must select the correct relation. Our new dataset makes use of 25 labels ranging from familial relations to professional relations.<\/p>\n
Finally, the Fantasy Reasoning task assesses a language model\u2019s ability to reason within situations that go against common sense or in some way violate the rules of the real world; humans do this easily, e.g., when reading a science fiction book. We collect a corpus of contexts that language models are extremely unlikely to be familiar with, paired with yes-no questions.<\/p>\n
References:<\/p>\n
Metaphor Understanding: https:\/\/github.com\/google\/BIG-bench\/tree\/main\/bigbench\/benchmark_tasks\/metaphor_understanding<\/a><\/p>\n Implicit Relations: https:\/\/github.com\/google\/BIG-bench\/tree\/main\/bigbench\/benchmark_tasks\/implicit_relations<\/a><\/p>\n