Within the field of Computational Creativity, evaluation is one of the most important and more difficult tasks. Sometimes evaluation is part of the creative systems themselves, becoming an internal evaluation. Being a module of a creative system, it is useful to evaluate how effective this internal evaluation is. In this paper, we propose a procedure for the (meta-) evaluation of internal evaluation modules, that allows for incremental development of both the evaluation module and the creative system, which are considered fully independent of each other. The procedure works by statistically comparing the evaluation of the aver- age output of the generation system with the best re- sults from the same, to see if the evaluation procedure can statistically distinguish the two. We then show how to apply the procedure giving one example evaluating a module we designed to assess structural coherence in generated folk music.

Meta-evaluating quantitative internal evaluation: a practical approach for developers

Filippo Carnovalini;Antonio Rodà;
2021

Abstract

Within the field of Computational Creativity, evaluation is one of the most important and more difficult tasks. Sometimes evaluation is part of the creative systems themselves, becoming an internal evaluation. Being a module of a creative system, it is useful to evaluate how effective this internal evaluation is. In this paper, we propose a procedure for the (meta-) evaluation of internal evaluation modules, that allows for incremental development of both the evaluation module and the creative system, which are considered fully independent of each other. The procedure works by statistically comparing the evaluation of the aver- age output of the generation system with the best re- sults from the same, to see if the evaluation procedure can statistically distinguish the two. We then show how to apply the procedure giving one example evaluating a module we designed to assess structural coherence in generated folk music.
Proceedings of the 12th International Conference on Computational Creativity
File in questo prodotto:
File Dimensione Formato  
ICCC_2021_paper_98.pdf

accesso aperto

Tipologia: Published (publisher's version)
Licenza: Creative commons
Dimensione 138.51 kB
Formato Adobe PDF
138.51 kB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

Caricamento pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11577/3406112
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact