Metric for evaluating mathematical models natural language words for machine translation

Abstract

The purpose of this study is to describe and analyze a new metric for evaluating the effectiveness of mathematical models of natural language words based on an extensible input language in machine translation. A new SSM (Structural Semantic Metric) metric has been developed and researched, the methodology of which includes a description (requirement, calculation principle, analytical model, calculation algorithm) and group experiments on six types of mathematical models of words in three natural languages (Uzbek, English, Russian). The metric is implemented in Python, the software products at the input accept a mathematical model represented as a string of characters of an extensible input language, and at the output numerical values for the formula parameters and a general metric estimate. The result of the metric is a numerical value in the range of 0 and 1. A gradation of degrees of reflection of the structural and semantic relationships of mathematical models relative to the range of 0 and 1 is recommended. The experiments obtained prove the high semantic adequacy of mathematical models with reference structures of words in each of the natural languages, which can provide a very high degree (90% - 95%) of translation of words used in machine translation texts between the above languages in 6 directions. The results also highlight the possibility of expanding the model base with new mathematical models of natural language words using this methodology.