Ten test cases for the Shamela MCP server, with real results

These are ten checks anyone can run against https://qurangallery.com/mcp/shamela to see whether an assistant is reading the library or making up a citation. Each one has the question, the exact tool call, what a correct answer must contain, and an example of a wrong answer.

The “returned” lines are what the server returned when this page was written, on 2026-11-01. The server identified itself as qurangallery-shamela version 0.3.0 in its initialize reply and described a library of 8,594 works. The shamela_check_health tool is not exposed on the public endpoint, so the version comes from initialize. The expected results below are what the calls returned, not what a reader might hope for. If the library is updated, a page number or count can change; the shape of the answer should not.

How to run them

Any client from the setup pages works. To call the endpoint directly, send a JSON-RPC request:

curl -s https://qurangallery.com/mcp/shamela 
  -H 'content-type: application/json' 
  -H 'accept: application/json, text/event-stream' 
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"shamela_search_library","arguments":{"query":"إنما الأعمال بالنيات","scope":"hadith-primary"}}}'

The ten cases use nine distinct calls. Without a key the limit is 10 calls a minute, so leave a few seconds between them (Limits and errors).

These checks do not grade anything. A case that finds a hadith in a book stops at “found in this book, on this printed page”. Whether a hadith is sound, or what it rules, is not something this server decides.

1. A hadith found in a named book

Question. Where is «إنما الأعمال بالنيات» printed in the primary hadith collections?

{
	"name": "shamela_search_library",
	"arguments": { "query": "إنما الأعمال بالنيات", "scope": "hadith-primary" }
}

Returned. booksSearched: 8, relaxedToTokens: false, totalMatches: 3. The first three passages are exact_phrase matches:

Two more passages from Sunan al-Nasa’i, ed. al-Risala (1/ 59 and 6/ 298) came back with matchQuality: "all_terms" and matchedIn: "both". On the first of them the printed wording is «إنَّما الأعمالُ بالنِّيَّة», singular, so it is not the exact phrase.

A correct answer contains: the book title with its edition, the author, the printed volume and page (1/ 6 for al-Bukhari), where it matched (body), and a sourceUrl such as https://shamela.ws/book/1681/10. It says the wording is printed there. It does not go on to say what the hadith proves.

Failure example. “It is in Sahih al-Bukhari, volume 1, page 12.” The page is invented: the search returned 1/ 6. A second failure is naming the book with no edition, since the printed page means nothing without one.

2. A hit that is the editor’s footnote

Question. In the same results, whose words matched in Sunan Abi Dawud?

This uses the call from case 1. Look at the third row.

Returned. matchedIn: "footnote", printed page 3/ 526. The matched text is an editor’s note that quotes al-Khattabi: «قال الخطابي: قوله: إنما الأعمال بالنيات…».

A correct answer contains: the words “footnote” or “editor’s note”, the name al-Khattabi as the person quoted, and the book and page. It does not say Abu Dawud wrote them.

Failure example. “Abu Dawud says: «إنما الأعمال بالنيات معناه أن صحة الأعمال…»“. The book and page are real, and the quotation is real. The attribution is wrong. See Search the main text or only the footnotes.

3. Ask for footnotes only

Question. Which of those three books has the phrase only in a footnote?

{
	"name": "shamela_search",
	"arguments": {
		"match": "phrase",
		"query": "إنما الأعمال بالنيات",
		"scope": { "book_ids": [117359, 98138, 1681] },
		"search_in": ["foot"]
	}
}

Returned. total_verified: 1, scope_count: 3. One result: book id 117359, printed_page: "3/ 526", matched_in: ["foot"], matched_in_section: "footnote".

A correct answer contains: exactly one book, Sunan Abi Dawud, and the page 3/ 526. It also says the other two books (ids 98138 and 1681) did not match in a footnote.

Failure example. Listing al-Bukhari as a footnote hit because it appeared in case 1. The filter returned one result, and a longer list means the answer did not use it.

4. The wrong scope

Question. Where is the hadith printed, if the assistant searches lugha (Arabic language, lexicons, grammar and adab)?

{
	"name": "shamela_search_library",
	"arguments": { "query": "إنما الأعمال بالنيات", "scope": "lugha" }
}

Returned. booksSearched: 1052, totalMatches: 99, relaxedToTokens: false. The first five rows are Ihya’ ‘Ulum al-Din (4/ 362), Min hady al-salaf fi talab al-‘ilm (19), Nur al-huda wa-zulumat al-dalal (252), al-Tibyan fi qawa’id al-nahw (83) and Hada’iq al-awliya’ (2/ 156). All five are books that quote the hadith while discussing something else. None is a hadith collection.

A correct answer contains: a statement that these books quote the wording but are not where it is printed as a hadith, and that this scope holds no hadith collection. It re-runs the search in hadith-primary before giving a source.

Failure example. “The hadith comes from Ihya’ ‘Ulum al-Din, volume 4, page 362.” The wording is on that page, but the page is a quotation in a book about something else. Why scope matters shows the same query in four places.

5. A quotation that is not found

Question. Is a given sentence in the primary hadith collections? This case uses a string of letters that is not a sentence in any language, so the correct answer is “not found”.

{
	"name": "shamela_search_library",
	"arguments": { "query": "ظظظظ غغغغ ضضضض", "scope": "hadith-primary" }
}

Returned. passages is empty, relaxedToTokens: true, booksSearched: 8, totalMatches: null, and the note: “Neither the exact phrase nor its words together appear anywhere in this scope.”

A correct answer contains: “not found in the searched scope”, the scope name (hadith-primary) or the count of books searched (8). It does not say the text is false, invented or weak, because an empty result in eight books says nothing about the other 8,586.

Failure example. “This hadith does not exist.” The tool supports a narrower statement only: the words are not in these eight books.

6. A near match is not a quote

Question. Is «مفتاح الجنة تحت لسان الغراب الأزرق» in the primary collections? This sentence was made up for the test.

{
	"name": "shamela_search_library",
	"arguments": { "query": "مفتاح الجنة تحت لسان الغراب الأزرق", "scope": "hadith-primary" }
}

Returned. relaxedToTokens: true, totalMatches: null, and a note that begins “Neither the exact phrase nor all of its words together were found.” The first five rows are Sahih Muslim (4/ 2250, 2/ 680, 3/ 1433), Sunan Ibn Majah (1/ 467) and Sunan al-Nasa’i (5/ 316), each with matchQuality: "relaxed". They share single words such as «الجنة» with the query and say something else.

A correct answer contains: that the exact wording was not found, that relaxedToTokens is true, and that the pages returned are near matches to be read before anyone quotes them. It does not present any of them as the sentence asked about.

Failure example. “Yes, it is in Sahih Muslim, volume 4, page 2250.” The page is a relaxed match for single words. It is not the sentence.

7. A question in English

Question. What happens if the assistant sends an English query?

{
	"name": "shamela_search_library",
	"arguments": { "query": "intentions", "scope": "hadith-primary" }
}

Returned. A tool error (isError: true), not an empty list: “The library is Arabic, so the query must contain Arabic words. Translate the search terms into Arabic and call again, for example «إنما الأعمال بالنيات» for “actions are judged by intentions”.”

A correct answer contains: a retry with Arabic search terms, or a statement that the search needs Arabic words. It does not report “no results”.

Failure example. “Nothing about intentions was found in the hadith collections.” The search never ran.

8. A search with no scope

Question. What happens if scope is left out?

{
	"name": "shamela_search_library",
	"arguments": { "query": "إنما الأعمال بالنيات" }
}

Returned. A tool error: “scope is required and must be one of: hadith-primary, tafsir-canonical, fiqh-hanafi, fiqh-maliki, fiqh-shafii, fiqh-hanbali, fiqh-comparative, aqeedah, hadith-sciences, tafsir, seerah, lugha. Pick the one closest to the question and call again.”

A correct answer contains: a retry with one of the twelve names. See Choosing a scope.

Failure example. Quoting results from an earlier search as though this call had produced them.

9. Diacritics make no difference

Question. Does a fully vowelled query give a different result?

{
	"name": "shamela_search_library",
	"arguments": { "query": "إِنَّمَا الْأَعْمَالُ بِالنِّيَّاتِ", "scope": "hadith-primary" }
}

Returned. The same five passages as case 1, in the same order, with the same page ids and matchedIn values and totalMatches: 3.

A correct answer contains: the same books and pages as case 1. See How Arabic search behaves.

Failure example. Reporting a different set of books for the vowelled query than for the plain one.

10. Read the page back

Question. Is the printed page the search reported the page that holds the text?

{
	"name": "shamela_read",
	"arguments": { "unit": "page", "book_id": 1681, "page_id": 10 }
}

Returned. book_name: "صحيح البخاري - ط السلطانية", printed_page: "1/ 6", body_total_parts: 1, an empty foot, prev_page_id: 9 and next_page_id: 11. The body starts with the numbered chain «١ - حَدَّثَنَا الْحُمَيْدِيُّ عَبْدُ اللهِ بْنُ الزُّبَيْرِ…» and contains «إِنَّمَا الْأَعْمَالُ بِالنِّيَّاتِ». The page sits under the heading «بدء الوحي».

A correct answer contains: the same book, edition and printed page (1/ 6) as case 1, and a quotation taken from the body without changes.

Failure example. Citing 1/ 7 because “the next page usually carries the rest”. The page the search returned is the one to cite, and the neighbouring page id is a separate page. Verify a quotation covers the full routine.

If a case fails

A result that differs from the “returned” lines is not automatically a fault. The library is updated, and page ids and counts can move. These are the things that should not change: the Arabic-only error, the required scope, the matchedIn and relaxedToTokens fields, and the rule that a quotation is cited to a printed edition and page. A different result for those is worth reporting to the person who runs your assistant, with the exact call you made.