SLUE test set evaluation isn’t publicly available to ensure a fair evaluation. Kindly submit the form, and our team will review it. Your submission will remain confidential.
Prepare a .tsv file with 4 columns as demonstrated in the illustration below. Please specify none, if the prediction does not exist. For example, Voxpopuli is for ASR and NER, it needs
pred_ner columns, and all entities in
pred_sentiment should be
None. If no NE found,
pred_ner should be
id pred_text pred_ner pred_sentiment id10012_0AXjxNXiEzo_00001 like i said less manicured in a good way i think i think that what you know people None Positive 20150518-0900-PLENARY-15-en_20150518-18:48:27_2 we all agreed at the last session in strasbourg that development is important but we need to remember it now when we are talking about the financial contributions [['GPE', 'strasbourg']] None 20180530-0900-PLENARY-26-en_20180530-20:01:52_0 madam president first of all i would like to thank all the members who have participated in this important debate for their different contributions. None None
Your submission must contain at least one of the SLU tasks other than ASR to be evaluated. We’ll still accept submissions with or without
pred_text if one of
pred_sentiment exists, but we will not evaluate your submission if you only submit
However, for example, if you only want to evaluate the sentiment analysis system, you could submit only the prediction result of the sentiment analysis system (
Most importantly, if you fail to include any of the columns in your submission, we won’t be able to calculate your SLUE score and thus cannot rank it.
Send tsv file to slue-committee “AT” googlegroups.com