cheRAGh Project
“We lit the lamp of search in the night of questions,
So the path of answers may emerge from its light through challenges”
Comprehensive Benchmark for Retrieval-Augmented Generation (RAG) Systems in Persian
cheRAGh is a collection of standard benchmarks for evaluating Retrieval-Augmented Generation (RAG) components and systems in the Persian language. By providing specialized datasets, standard evaluation metrics, and leaderboards, this project offers a unified platform for comparing, analyzing, and developing Persian RAG models and systems.
The goal of cheRAGh is to create an open-source, transparent, and reproducible reference for researchers, developers, and organizations looking for a fair evaluation of retrieval models, Embedding models, Reranking models, and other RAG system components.
By covering various domains including Legal, Religious, Scientific, Educational, and General, cheRAGh strives to provide an accurate picture of model performance in real-world applications and enable their comparison based on standard criteria.
What is cheRAGh and what is its use?
Retrieval-Augmented Generation (RAG) systems consist of several independent components such as Embedding models, Retrieval, Reranking, and Large Language Models (LLMs). The quality of the final response of these systems depends on the performance of each of these components; therefore, their fair evaluation is only possible if it is based on standard datasets, clear metrics, and a reproducible process.
cheRAGh was developed with the aim of creating a comprehensive and standard reference for evaluating Persian RAG systems. This project provides a set of independent and complementary benchmarks, each evaluating one of the main components of a RAG system. All evaluations are based on Persian datasets, standard metrics, and transparent processes to ensure the results are reliable, comparable, and reproducible.
Currently, the first benchmark of the cheRAGh project, specifically for evaluating Embedding models, has been published and is available to researchers and developers via the Embedding Benchmark page. This benchmark compares the performance of different models in terms of information retrieval quality in Persian. Other cheRAGh benchmarks, including the evaluation of Reranker models, Retrieval systems, Large Language Models (LLMs), and other components of the RAG pipeline, are also under development and will be gradually added to the collection.
Covered Domains
To ensure the evaluations represent real-world applications, cheRAGh datasets cover five important domains:
Legal
Laws, regulations, and legal documents.
Religious
Islamic texts and religious questions.
Scientific
Abstracts and academic papers texts.
Educational
Textbooks and educational resources.
General
Everyday user questions and interactions.
The selection of these domains was made with the intention of covering a diverse range of writing styles, vocabularies, concept complexities, and informational needs in the Persian language. Thus, the cheRAGh results do not merely indicate a model’s performance in a single specific field, but provide a more comprehensive picture of its capabilities across various applications.
Furthermore, cheRAGh uses evaluation metrics tailored to each specific task for every benchmark. For example, in the Embedding benchmark, the Mean Reciprocal Rank (MRR) and Recall@5 metrics are utilized to measure the quality of ranking and the ability to retrieve correct documents. In other benchmarks, specialized metrics suited to the nature of the task will be employed to ensure each component is evaluated based on indicators that provide the most accurate picture of its real-world performance.
cheRAGh provides an ideal platform for researchers, developers, and organizations to compare different models and systems using a standardized and reliable method, identify their strengths and weaknesses in various domains, and track their progress over time. The ultimate goal of this project is to establish an authoritative reference for evaluating Persian RAG systems and to aid in the development of AI research and products based on the Persian language.
cheRAGh Project Benchmarks
cheRAGh offers a suite of independent and complementary benchmarks, each evaluating one of the main stages of a Retrieval-Augmented Generation (RAG) system. Each benchmark has its own dedicated datasets, evaluation metrics, and leaderboard, and is published and updated independently. Together, these benchmarks provide a comprehensive picture of Persian RAG systems’ performance, from information retrieval to final response generation.
This benchmark evaluates the quality of embedding models in semantically representing Persian texts and retrieving relevant documents. Models are tested on cheRAGh’s multi-domain datasets, and their performance is reported using metrics such as MRR and Recall@5.
Status: ✅ Published
View Benchmark:
https://huggingface.co/spaces/QomSSLab/cheRAGh-Embedding
After the initial retrieval, reranking models are responsible for reordering the retrieved documents based on their relevance to the query. This benchmark evaluates the ability of Reranker models to improve retrieval result quality and increase RAG system accuracy.
Status: 🚧 Under Development
In many RAG systems, the language model must be able to select and call appropriate tools to answer queries. This benchmark evaluates models’ ability to choose the correct tool, generate a valid call, use the tool’s output, and continue the reasoning and answering process.
Status: 🚧 Under Development
This benchmark evaluates the complete performance of RAG systems from receiving a query to generating the final response. In this section, all system components, including retrieval, reranking, tool usage, and response generation, are assessed cohesively to determine the final system quality in real-world scenarios. This benchmark enables direct comparison of different RAG systems and provides a comprehensive view of their practical application performance.
Status: 🚧 Under Development
These four benchmarks cover the core stages of a RAG system and together provide a comprehensive platform for evaluating and comparing Persian RAG models and systems. As the cheRAGh project evolves, new datasets, evaluation metrics, additional models, and analytical reports will be added to each of these benchmarks.