Allen Institute for AI:Shared Task on Word Embedding Evaluation for Ancient and Historical Languages
Allen Institute for AI at SIGTYP 2024 Shared Task on Word Embedding Evaluation for Ancient and Historical Languages
Authors: Lester James V. Miranda
Abstract: In this paper, we describe Allen AI’s submission to the constrained track of the SIGTYP 2024 Shared Task. Using only the data provided by the organizers, we pretrained a transformer-based multilingual model, then finetuned it on the Universal Dependencies (UD) annotations of a given language for a downstream task. Our systems achieved decent performance on the test set, beating the baseline in most language-task pairs, yet struggles with subtoken tags in multiword expressions as seen in Coptic and Ancient Hebrew. On the validation set, we obtained ≥70% F1-score on most language-task pairs. In addition, we also explored the cross-lingual capability of our trained models. This paper highlights our pretraining and finetuning process, and our findings from our internal evaluations.
#SIGTYP2024