إسناد
CS619 Final Year Project — Complete Student Guide

IsnadGuard
Hadith Chain Defect Detection System

From understanding the Islamic domain and computer science foundations, to downloading your dataset, designing your system, and submitting all four assignments — this guide covers everything.

22Chapters
8Project Modules
4Assignments
577KIsnad Chains
Chapter 1 — Domain Understanding

What is a Hadith?

A Hadith (حديث — plural: Ahadith) is a recorded account of something the Prophet Muhammad ﷺ said, did, or silently approved of. After the Holy Quran, the Ahadith form the second most important foundation of Islamic guidance and law. For over fourteen centuries, Muslim scholars have treated the verification of these narrations as both a scholarly duty and a religious obligation — because a fabricated or distorted statement attributed to the Prophet ﷺ would constitute a grave religious harm.

Every Hadith you encounter in any of the six major Hadith collections — Sahih al-Bukhari, Sahih Muslim, Sunan Abu Dawud, Jami al-Tirmidhi, Sunan al-Nasa'i, and Sunan Ibn Majah — has two parts:

  • Isnad (إسناد) — The chain of narrators who transmitted the Hadith from one generation to the next, tracing back to someone who heard or witnessed it directly.
  • Matn (متن) — The actual content — what the Prophet ﷺ said or did.
إِنَّمَا الأَعْمَالُ بِالنِّيَّاتِ Example of Matn: "Actions are judged by their intentions." — This is the text of the very first Hadith in Sahih al-Bukhari, accompanied by a full chain of five narrators traced back to Umar ibn al-Khattab, a companion of the Prophet ﷺ.

How Scholars Grade a Hadith

Not every statement attributed to the Prophet ﷺ is authentic. Over the centuries, some narrations were fabricated by people with political or religious agendas, and others were simply misremembered or confused. Islamic scholars developed a rigorous science called Ilm al-Hadith — the science of Hadith criticism — to evaluate each narration systematically. The grade assigned to a Hadith depends primarily on the quality of its narrator chain:

GradeArabicMeaningWhat it implies for the chain
SahihصحيحAuthenticUnbroken chain, every narrator reliable and precise in memory, no hidden defects
HasanحسنGoodMeets most criteria; one narrator is slightly weaker in precision than ideal
Da'ifضعيفWeakChain has at least one structural defect or one unreliable narrator
Mawdu'موضوعFabricatedA known liar appears in the chain — the Hadith is completely rejected
💡
Your project's role: IsnadGuard is a computational tool that analyzes the structural properties of narrator chains to assist in identifying defects. You are not replacing the scholarly tradition — you are building software that makes a part of their analysis process faster and more transparent by using machine learning and explainability methods. This is a legitimate and active area of research in digital Islamic humanities.
Think About This
  1. In your own words, explain to a non-Muslim friend what an Isnad is and why it matters, using an analogy from software engineering — for example, a version control history or an audit log.
  2. Why would a "fabricated" Hadith be more harmful than simply a "weak" one? What does this tell you about why detection matters?
Chapter 2 — Domain Understanding

The Isnad — Understanding the Chain Structure

The word Isnad (إسناد) comes from the Arabic root meaning "to support" or "to lean upon." It is the ordered sequence of people who transmitted a Hadith from the Prophet ﷺ forward through time until it was recorded in a book. Each person in the chain heard the narration from the person before them and passed it to the person after them.

In computational terms, you can think of an Isnad as a directed linked list — each narrator is a node, each transmission is an edge pointing from the source toward the recorder. The list must be complete from beginning to end. A missing node anywhere in the chain is a structural defect.

How a Chain Looks

The chain below represents a simplified version of a real Isnad. Reading it left to right, it shows how the narration was passed forward in time:

Prophet ﷺSource — Gen 0
→
Umar ibn al-KhattabCompanion — Gen 1
→
Alqama ibn WaqqasSuccessor — Gen 2
→
Muhammad ibn IbrahimGen 3
→
Yahya ibn Sa'idGen 5
→
Imam al-BukhariCollector — Gen 9

Each arrow means "transmitted to." Bukhari heard from Yahya, who heard from Muhammad ibn Ibrahim, and so on back to the Companion who heard from the Prophet ﷺ directly.

Narrator Generations — Tabaqat

Every narrator belongs to a historical generation called a Tabaqah (طبقة — plural: Tabaqat). The generation number is absolutely critical for your defect detection system. Two consecutive narrators in a chain must belong to adjacent or near-adjacent generations — because they must have been alive at the same time to transmit to each other. If your data shows that Narrator A is from Generation 3 and Narrator B, who supposedly heard directly from A, is from Generation 7, that four-generation gap is your primary signal that someone is missing between them.

GenerationArabic TermDescription
0—Prophet Muhammad ﷺ — the ultimate source
1Sahaba (صحابة)Companions — personally met the Prophet ﷺ
2Tabi'un (تابعون)Successors — met the Companions
3Tabi al-Tabi'inSuccessors of Successors
4–9Later generationsTransmission through subsequent scholarly generations
~9–12CollectorsMajor Hadith book authors — Bukhari, Muslim, Abu Dawud, etc.
Think About This
  1. A Companion (Generation 1) lived in the 7th century CE. A narrator from Generation 7 would have lived approximately 150–200 years later. Is it possible for Generation 7 to have heard directly from Generation 1 without anyone in between? What does this tell your system?
  2. Before you start coding anything, sketch a paper diagram of how you would represent a narrator chain as a data structure in Python — what type would the chain be, and what fields would each narrator object contain?
Chapter 3 — Domain Understanding

Types of Chain Defects — What Your System Must Detect

A defect in an Isnad is called an Illah (عِلَّة) — a hidden weakness. Islamic scholars spent centuries classifying these defects and developing systematic methods to identify them. Your project takes this classical taxonomy and implements it computationally. There are five main defect types your system must handle:

Munqati'
مُنقَطِع Broken / Disconnected

One narrator is missing from somewhere in the middle of the chain. Detected by finding a generation gap of exactly one between two consecutive narrators — meaning one person who should appear between them has been omitted.

Mursal
مُرسَل Slipped / Sent

A Successor (Generation 2) narrates directly from the Prophet ﷺ, skipping the Companion (Generation 1) entirely. The very first link is missing. Detected by checking whether any Generation 1 narrator appears in the chain.

Mu'dal
مُعضَل Troubled / Problematic

Two or more consecutive narrators are missing — the generation gap between two narrators is two or more. This is a more severe form of Munqati'. Both must be detected separately, as they have different implications.

Mawquf
مَوقُوف Stopped

The chain stops at a Companion and does not reach back to the Prophet ﷺ — the narration is attributed to the Companion's own statement. Detected by checking whether the final narrator in the chain is a Generation 1 figure.

Weak Narrator
ضعف الراوي Narrator Weakness

The chain is structurally complete but contains a narrator whose reliability grade is Da'if (weak), Majhul (unknown), or lower. The problem is not the structure but the trustworthiness of a person within it.

Example — A Broken Chain (Munqati')

Prophet ﷺGen 0
→
CompanionGen 1
→
Narrator BGen 3
✗
MISSINGGen 4, 5, 6?
✗
Narrator C (Da'if)Gen 7
→
CollectorGen 9

The gap from Gen 3 to Gen 7 signals Mu'dal (four generations missing). Narrator C's Da'if grade signals an additional weak narrator defect. Both are present in the same chain.

Think About This
  1. If a chain has both a Munqati' gap and a weak narrator, which defect is more serious? How would your system communicate both findings to the user?
  2. For your SRS functional requirements, write the requirement for Munqati' detection using the correct format: "The system shall..." — without mentioning Python or any library name.
Chapter 4 — CS Domain Background

Machine Learning — The Core CS Discipline Behind This Project

Machine Learning (ML) is a branch of computer science in which systems learn patterns from data rather than following explicitly programmed rules. Instead of a programmer writing every decision, a machine learning model is trained on examples and learns to generalize from them. In the context of your project, you are training a model to look at the features of a narrator chain and predict its grade — Sahih, Hasan, or Da'if.

The Three Types of Machine Learning

Your project uses supervised learning, so understanding how the three types differ gives you the necessary context to explain your choice in the SRS:

TypeDescriptionUsed in this project?
Supervised LearningThe model is trained on labeled data — each example has a known correct answer. The model learns the relationship between input features and labels.Yes — chain grades (Sahih/Da'if) are the labels
Unsupervised LearningNo labels — the model finds hidden patterns or groupings in the data on its own (e.g., clustering).Optionally, for clustering narrator types
Reinforcement LearningAn agent learns by interacting with an environment and receiving rewards or penalties.Not used in this project

Classification — What Your ML Module Does

Within supervised learning, your task is a classification problem — assigning each chain to one of several categories. The algorithm looks at a set of input features you provide (numbers derived from the chain structure) and maps them to an output category (the chain's grade).

The Algorithms You Will Compare

Your Module 6 requires you to train and compare at least two classification algorithms using the Scikit-learn library. Here is what each one does conceptually:

AlgorithmHow it works conceptuallyStrengths for this project
Decision TreeBuilds a tree of if/then rules from the training data. Each branch represents a decision based on one feature value.Very interpretable — you can read the tree and understand exactly what rules it learned. Good for a first model.
Random ForestBuilds hundreds of decision trees, each trained on a slightly different random subset of the data, and takes the majority vote.Much more accurate than a single tree. Resistant to overfitting. Also provides feature importance scores.
Logistic RegressionDespite the name, it is a classification algorithm. It calculates the probability that a chain belongs to each grade class based on a weighted combination of its features.Fast, efficient, good baseline. Works well when features are reasonably linear.

Key Evaluation Metrics You Must Know

After training, you measure how well your model performs on data it has never seen before (the test set). Your SRS should mention these metrics as non-functional requirements, and your Design Document must show them in your test cases:

MetricWhat it measuresPlain-English meaning
AccuracyOverall correct predictions / total predictionsOut of all chains I classified, what percentage did I get right?
PrecisionTrue positives / (true positives + false positives)Of the chains I labeled Da'if, how many were actually Da'if?
RecallTrue positives / (true positives + false negatives)Of all the truly Da'if chains, how many did I catch?
F1 ScoreHarmonic mean of Precision and RecallA balanced measure when your classes are unequal in size
Confusion MatrixA table showing correct vs. incorrect predictions per classShows exactly which classes are being confused with which others
Think About This
  1. If your model has 90% accuracy but only detects 40% of the actually Da'if chains, which metric is more important for your use case — accuracy or recall? Explain why.
  2. Why do you need to split the dataset into training and testing sets? What would happen if you trained and tested on the same data?
  3. In your Design Document Test Cases section, write one test case for the ML classifier. What is the input, what is the expected output, and what metric confirms success?
Chapter 5 — CS Domain Background

Data Science — The Pipeline That Connects Everything

Data Science is the discipline of extracting meaningful insights from structured and unstructured data through a combination of statistics, programming, and domain knowledge. It is not a single technique but a pipeline — a sequence of steps that transforms raw data into useful knowledge. Your entire project follows this pipeline from beginning to end.

The Data Science Pipeline in Your Project

Every module you build corresponds to a stage in this pipeline. Understanding the pipeline as a whole helps you see why the modules are ordered the way they are, and why skipping a step causes problems downstream:

Pipeline StageYour Project ModuleWhat goes wrong if you skip it
Data CollectionModule 1 — loading hadith-kgYou have no data to work with
Data CleaningModule 1 — handling nulls, standardizing gradesMissing values crash your algorithms; inconsistent labels produce wrong classifications
Exploratory Data AnalysisModule 5 — statistics and chartsYou build a model without understanding your data — you cannot explain your results in the viva
Feature EngineeringModule 6 — extracting numerical features from chainsThe ML model has nothing meaningful to learn from — raw names are not numbers
ModelingModule 6 — training classifiersNo predictive capability
EvaluationModule 6 — accuracy, F1, confusion matrixYou cannot claim your model works; your test cases have no basis
InterpretationModule 7 — SHAP explainabilityYou have a black-box prediction with no explanation — useless for scholars
DeploymentModule 8 — Streamlit dashboardYour system only runs in a notebook; no real user can use it

Feature Engineering — The Most Important Step You Will Do

Machine learning algorithms work with numbers, not names or text. The process of converting raw data into meaningful numerical inputs for a model is called feature engineering, and it is arguably the most important skill in data science. For this project, you must convert each narrator chain into a set of numerical features. Here are the features you should design:

Feature NameHow to Compute ItWhy It Matters
chain_lengthCount the number of narrators in the chainUnusually short chains may signal missing narrators
max_gen_gapLargest generation difference between any two consecutive narratorsDirectly signals Munqati' or Mu'dal severity
total_gen_gapsCount of all consecutive pairs with a generation gap > 1Measures how many breaks exist, not just the largest one
weak_narrator_countNumber of narrators whose grade is Da'if, Majhul, or worseDirectly signals the weak narrator defect
min_grade_scoreEncode grades as numbers (Thiqah=3, Hasan=2, Da'if=1, Majhul=0, etc.) and take the minimum across the chainThe weakest link determines the chain's overall quality
has_companionBinary: 1 if a Generation 1 narrator is in the chain, 0 if notAbsence indicates Mursal defect
ends_at_companionBinary: 1 if the last narrator is Generation 1, 0 if notIndicates Mawquf — chain does not reach the Prophet ﷺ
Think About This
  1. Can you think of one additional feature that might help the ML model — something not listed in the table above? Explain your reasoning.
  2. If the min_grade_score feature is 0 (Majhul) but the chain has no structural gaps, what grade would you expect the model to predict? Why?
  3. For your Data Pipeline Diagram in the Design Document, draw the flow from the raw database through all the steps above to the final prediction. How many steps are in the pipeline?
Chapter 6 — CS Domain Background

Explainable AI — Why It Matters More Here Than Anywhere

Explainable AI, commonly abbreviated as XAI, is a field of artificial intelligence focused on making the decisions of machine learning models understandable to humans. When a model classifies an image as a cat, most users are satisfied with that answer. But when a model says "this Hadith chain is Da'if," a scholar, a researcher, or anyone taking that verdict seriously will immediately ask: why? Which part of the chain caused this classification? Was it a weak narrator, a broken link, or something else?

Without an explanation, your model is a black box — it produces an answer but cannot justify it. In the domain of Hadith authentication, an unjustified verdict is essentially useless, because the entire tradition of Hadith science is built on transparency of reasoning. This is precisely why the XAI module is not optional in your project — it is what makes the system academically credible and practically meaningful.

SHAP — The Method You Will Use

SHAP stands for SHapley Additive exPlanations. It is a mathematically grounded method for explaining individual predictions made by any machine learning model. The core idea comes from game theory: each feature of your input (each number in the chain's feature vector) gets a "Shapley value" that represents how much it contributed to pushing the prediction in a particular direction.

Concretely, for a chain that your model classifies as Da'if, SHAP might produce an explanation like: "The max_gen_gap feature contributed +0.42 toward the Da'if prediction, while weak_narrator_count contributed +0.31, and chain_length contributed -0.08 (slightly away from Da'if)." This tells a scholar exactly what structural feature was most responsible for the verdict.

📌
For your Design Document: SHAP is not a separate model — it is an analysis layer that wraps your already-trained classifier. You train the classifier in Module 6, then in Module 7 you create a SHAP Explainer object, pass it your trained model, and call it on individual chain feature vectors to get the explanation values. The output is a set of bar charts or waterfall charts showing feature contributions per prediction.
Think About This
  1. If SHAP tells you that has_companion = 0 (absence of a Companion) was the most important feature for a Da'if prediction, which defect type does that correspond to? What would you tell the user on screen?
  2. Why is SHAP particularly important in the context of Hadith authentication compared to, say, spam email detection?
Chapter 7 — CS Domain Background

Graph Theory — How Narrator Chains Become Visual Networks

Graph theory is a branch of mathematics and computer science dealing with structures made up of nodes (also called vertices) connected by edges. Many real-world problems map naturally to graphs: social networks, road maps, web links, electrical circuits, and — as your project demonstrates — narrator transmission chains.

How a Narrator Chain Becomes a Graph

In your project, every narrator is a node and every transmission relationship (A heard from B) is a directed edge. The chain from the Prophet ﷺ through multiple narrators to the collector is a directed path graph — a sequence of nodes where edges only go in one direction. Your visualization module renders this graph on screen, coloring each node by the narrator's reliability grade and marking broken links with a distinct visual style.

The Library You Will Use

NetworkX is the standard Python library for creating, analyzing, and drawing graphs. It supports directed and undirected graphs, weighted edges, and various layout algorithms for positioning nodes on screen. For rendering, it works together with Matplotlib. You do not need to implement graph theory from scratch — NetworkX handles the data structure and Matplotlib handles the visual output.

What Graph Features Are Useful for Your Project

Beyond visualization, graph analysis can contribute to your ML feature set. Concepts like node degree (how many chains a narrator appears in), betweenness centrality (how many chains pass through a narrator), and connected components (whether the chain forms one continuous path) are all computable from graph structures and may be worth exploring as additional features in Module 6.

Think About This
  1. In your class diagram for the Design Document, which class is responsible for building the graph object from the chain data? What methods would it need?
  2. A narrator who appears in 5,000 different chains has a high node degree. Does that make them more or less likely to be reliable? Think about this before deciding whether to include it as a feature.
Chapter 8 — Dataset

Which Datasets to Use and Why

All datasets recommended for this project are peer-reviewed, publicly available, and free. No unofficial sources, no Kaggle datasets from anonymous contributors, and no data that requires any payment. The following three are your primary options:

DatasetSizeFormatBest For
hadith-kg (emadjumaah)577K chains, 49K narrator biographiesSQLite database (.db)Primary dataset — narrator grades already labeled, generation numbers included, chains fully linked. Use this for everything.
Sanadset 650K (arbml)650,986 Hadith recordsParquet / CSVAdditional chain records for testing. Use this if you want to test your detection system on a larger volume of chains.
MIS Dataset (Sahih Muslim)7,748 Hadith, 77,797 narrator linksExcel (.xlsx)Focused, clean dataset for Sahih Muslim specifically. Useful for prototyping and initial testing.
📖
How to cite these in your SRS and Design Document:
Jumaah, E. (2024). hadith-kg: A Knowledge Graph of the Sciences of Hadith. Hugging Face Datasets. DOI pending — cite as: https://huggingface.co/datasets/emadjumaah/hadith-kg

Al-Thubaiti, K. et al. (2022). Sanadset 650K: Data on Hadith Narrators. Data in Brief, 44, 108540. DOI: 10.1016/j.dib.2022.108540

Key Tables in hadith-kg

TableWhat It ContainsKey Columns
rawisNarrator biographies — the Rijal dataid, name, grade, tabaqah, death_year, is_stub
sanadsComplete isnad chains with overall gradeid, hadith_id, grade
sanad_rawisJunction — maps narrators to chains with position ordersanad_id, rawi_id, pos
hadithsHadith text (matn) and collection referenceid, collection_id, hadith_no, matn
collectionsThe Hadith books (Bukhari, Muslim, etc.)id, name, author
Chapter 9 — Dataset

Using Hugging Face — What It Is and How to Get Started

Hugging Face is the world's largest open-source platform for sharing AI models and datasets. Think of it as GitHub, but purpose-built for machine learning. The platform hosts over 500,000 public datasets, all freely accessible. Your primary dataset — hadith-kg — is hosted there.

Creating Your Account

1

Register at huggingface.co

Go to https://huggingface.co and click Sign Up. You can register with any email address — Gmail, Yahoo, or your VU email. No payment is required. A free account gives you full access to all public datasets.

2

Browse the Dataset

After logging in, go directly to https://huggingface.co/datasets/emadjumaah/hadith-kg — or search "hadith-kg" in the Datasets section. The Dataset Card shows a description and the available files. Click the "Files and versions" tab to see the actual files.

3

Generate an Access Token

For downloading through Python, you need a read-access token. Go to your account Settings → Access Tokens → New Token. Name it "isnadguard" and select "Read" permission. Copy the token and keep it safe — you will use it once during the download setup.

🌐
Use the Dataset Viewer first. Before you download anything, use the Dataset Viewer on the Hugging Face website to browse actual rows from the database. This costs zero bytes of download and gives you a clear picture of what the data looks like before you write a single line of code. This step is strongly recommended as your first action.
Chapter 10 — Dataset

Downloading: The Windows Method

If you prefer a graphical, point-and-click approach to downloading the dataset file, this is the method for you. It requires no programming, just a browser and the Hugging Face website.

1

Log in and Navigate to the Dataset

Go to https://huggingface.co/datasets/emadjumaah/hadith-kg in your browser while logged into your Hugging Face account.

2

Click "Files and versions"

On the dataset page, click the "Files and versions" tab. You will see a list of files in the repository. The file you need is hadith-kg.db — approximately 1.6 GB in size.

3

Click the Download Icon

Next to hadith-kg.db, click the download icon (↓). Your browser will begin downloading the file to your Downloads folder. Do not close the browser tab until the download completes.

4

Move the File to Your Project Folder

After downloading, move hadith-kg.db to a folder named data inside your project directory. Keep a backup copy somewhere safe — perhaps an external drive or Google Drive — so you never have to download it again.

⚠️
Important for slow or unstable connections: The Windows browser download method does NOT support resuming from where it stopped if your connection drops. If your internet is interrupted during download, the partial file is usually discarded and you must start from the beginning. If your connection is unstable, use the Python method in the next chapter instead — it supports resumable downloads.
Chapter 11 — Dataset

Downloading: The Python Code Method

Downloading through Python gives you more control, better progress visibility, and — most importantly — the ability to resume an interrupted download. This is the recommended method for most students.

Step 1 — Install the Required Library

Open your terminal or command prompt and run this command once. You do not need to run it again after it succeeds:

TERMINAL — Install huggingface_hub
pip install huggingface_hub

Step 2 — Log In to Hugging Face

Run this command in your terminal. It will ask you to paste the access token you generated in Chapter 9. You only need to do this once — your credentials are saved to your machine after the first login:

TERMINAL — Login once
huggingface-cli login

Step 3 — Download the Dataset File

Create a new Python file in your project folder — for example, download_dataset.py — and write the following code exactly as shown. Run it with python download_dataset.py in your terminal:

PYTHON — download_dataset.py
from huggingface_hub import hf_hub_download
import os

# Create the data folder if it does not exist
os.makedirs("./data", exist_ok=True)

print("Starting download of hadith-kg.db ...")
print("File size: approximately 1.6 GB. This may take a while.")
print("If interrupted, re-run this script — the download will resume.")

file_path = hf_hub_download(
    repo_id="emadjumaah/hadith-kg",
    filename="hadith-kg.db",
    repo_type="dataset",
    local_dir="./data",
    local_dir_use_symlinks=False
)

print(f"Download complete. File saved at: {file_path}")
print(f"File size on disk: {os.path.getsize(file_path) / (1024**3):.2f} GB")
✅
The local_dir parameter is what makes resuming possible. When you set a local directory, the Hugging Face library saves a partial file with a .incomplete extension. If your connection drops and you re-run the script, the library detects this partial file and resumes from where it stopped — you do not lose any progress. This is the key advantage over a browser download.

Downloading the Sanadset 650K (Alternative / Additional)

To download the Sanadset dataset instead or in addition, replace the repo_id and filename in the script above with the following values. Sanadset is stored in Parquet format, which is more efficient than CSV for large datasets:

PYTHON — Download Sanadset
from huggingface_hub import snapshot_download

# This downloads all Parquet files of the Sanadset into ./data/sanadset/
snapshot_download(
    repo_id="arbml/Sanadset",
    repo_type="dataset",
    local_dir="./data/sanadset",
    local_dir_use_symlinks=False
)
Chapter 12 — Dataset

Resumable Downloads — What to Do When the Connection Drops

Internet connections in Pakistan are often unstable, and a 1.6 GB file can take a long time to download. Here is exactly what happens in each scenario and what you should do:

ScenarioWhat happensWhat to do
Browser download interruptedPartial download is usually discarded. The file in your Downloads folder is incomplete and unusable.Delete the partial file and restart the download from the beginning. Consider switching to the Python method.
Python script interruptedA .incomplete partial file is saved in ./data/. No progress is lost.Re-run the same Python script. The library will detect the partial file automatically and resume from where it stopped.
Python script completes but file seems wrongThe download may have been corrupted partway through without the library detecting it.Check the file size in the print output — it should be approximately 1.60–1.65 GB. If it is much smaller, delete the .db file (not the .incomplete file) and re-run the script.

Using the Streaming Option — No Download Required

If downloading is genuinely impossible on your connection, you can stream just the records you need directly from Hugging Face into your Python session — no file is saved to disk at all. This is suitable for the Sanadset dataset but not for hadith-kg (which is a SQLite file and must be opened locally). Here is how streaming works for Sanadset:

PYTHON — Stream Sanadset without downloading
from datasets import load_dataset

# Install the datasets library first: pip install datasets
# streaming=True means no file is downloaded — data streams record by record

ds = load_dataset(
    "arbml/Sanadset",
    streaming=True,
    split="train"
)

# Take only the first 1000 records for your prototype work
sample = list(ds.take(1000))

import pandas as pd
df = pd.DataFrame(sample)
print(df.head())
print("Columns:", df.columns.tolist())
print(f"Loaded {len(df)} records")
⚠️
Streaming limitation: Every time you run your code in streaming mode, it fetches records from the internet again. This means you need an internet connection every time you run your project. For your viva demonstration, it is much safer to have the database file downloaded locally. Use streaming only during development when you need quick tests.
Chapter 13 — Dataset

Exploring the Dataset — What to Do Before Writing Any Module

The first thing you must do after successfully connecting to hadith-kg.db is explore the data. Do not jump to building the defect detector until you have spent time understanding what is actually in the database. Here is the exploration code to run:

PYTHON — explore_dataset.py — Run this first
import sqlite3
import pandas as pd

# Open connection to the database
conn = sqlite3.connect("./data/hadith-kg.db")

# ── Step 1: List all tables ──────────────────────────────
tables = pd.read_sql_query(
    "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name",
    conn
)
print("=== Tables in database ===")
print(tables)

# ── Step 2: See all narrator grade types ─────────────────
grades = pd.read_sql_query(
    "SELECT grade, COUNT(*) as count FROM rawis GROUP BY grade ORDER BY count DESC",
    conn
)
print("\n=== Narrator Grade Distribution ===")
print(grades)

# ── Step 3: See generation distribution ──────────────────
tabaqat = pd.read_sql_query(
    "SELECT tabaqah, COUNT(*) as count FROM rawis WHERE tabaqah IS NOT NULL GROUP BY tabaqah ORDER BY tabaqah",
    conn
)
print("\n=== Generation (Tabaqah) Distribution ===")
print(tabaqat)

# ── Step 4: Sample a complete chain ──────────────────────
chain_query = """
    SELECT sr.pos, r.name, r.grade, r.tabaqah, r.death_year
    FROM sanad_rawis sr
    JOIN rawis r ON sr.rawi_id = r.id
    WHERE sr.sanad_id = 1
    ORDER BY sr.pos ASC
"""
chain = pd.read_sql_query(chain_query, conn)
print("\n=== Sample chain — sanad_id = 1 ===")
print(chain)

# ── Step 5: Check chain length distribution ───────────────
lengths = pd.read_sql_query(
    "SELECT sanad_id, COUNT(*) as length FROM sanad_rawis GROUP BY sanad_id ORDER BY length DESC LIMIT 20",
    conn
)
print("\n=== Top 20 longest chains ===")
print(lengths)

conn.close()
print("\nExploration complete.")
🚫
Before you proceed to Module 1: You must be able to run the exploration script and explain what every line of output means. Your supervisor will ask you during the viva what you found in the data. If you cannot describe the grade distribution, the typical chain length, and what the tabaqah column contains — you have not explored the data properly.
Chapter 14 — System Design

System Inputs and Outputs — What the User Gives and Gets

Before designing any class or writing any logic, you must know precisely what the user interacts with. This section describes the inputs and outputs for every major module of your project. This directly informs your Use Case Diagram in the SRS and your Sequence Diagrams in the Design Document.

Module 1 — Data Loading

User Input

  • Path to the local hadith-kg.db file, entered through a file-selection button
  • Or: choice to use streaming (if no local file)

System Output

  • Summary panel: total narrators loaded, total chains loaded, grade distribution
  • List of any records with missing generation numbers or grades
  • Confirmation that data is ready for all other modules

Module 3 — Defect Detection

User Input

  • A chain ID (Sanad ID) — a number — entered in a search box
  • Or: selection from a browsable list of all chains

System Output

  • Ordered list of narrators with name, grade, and generation for each
  • Defect report: which of the five defect types were found
  • For each defect: position in chain and plain-English explanation
  • Chain visualization as a graph diagram
  • Overall verdict: Structurally Sound / Defective

Module 6 — ML Classifier

User Input

  • A chain ID to classify
  • Selection of model (Decision Tree / Random Forest / Logistic Regression)

System Output

  • Predicted grade (Sahih / Hasan / Da'if)
  • Confidence percentage for the prediction
  • Comparison with the classical scholarly grade stored in the database

Module 7 — XAI Explanation

User Input

  • Same chain ID that was classified in Module 6

System Output

  • Feature importance bar chart for this specific chain
  • Which feature contributed most to the prediction
  • Written plain-English explanation linking the dominant feature back to the defect it represents

Module 8 — Final Dashboard

User Input

  • A chain ID entered in a search box, or selected from a list

System Output

  • All outputs from Modules 3, 4, 6, and 7 in a single integrated view
  • One-click button to export everything as a PDF report
Chapter 15 — System Design

Implementation Logic — Pseudocode for Core Functions

This chapter gives you the logical skeleton of each core function as pseudocode. Pseudocode describes what the function should do in plain English, without being tied to any specific programming language. Your task is to take this pseudocode and implement it yourself in Python, making decisions about data types, error handling, and edge cases along the way.

Building a Chain from the Database

PSEUDOCODE — build_chain(chain_id)Your first task — implement this before any detection
FUNCTION build_chain(chain_id):

    -- Query the database --
    -- JOIN sanad_rawis with rawis WHERE sanad_id = chain_id --
    -- SELECT pos, name, grade, tabaqah, death_year --
    -- ORDER BY pos ASC --

    IF no rows returned:
        raise an error: "Chain ID not found"

    FOR EACH row in results:
        create a narrator object with: name, grade, tabaqah, death_year, position

    RETURN the ordered list of narrator objects

    -- Edge cases to handle in your implementation: --
    -- What if tabaqah is NULL for some narrators? --
    -- What if grade is NULL? --
    -- What if chain_id is not an integer? --

Munqati' Detection — Broken Chain

PSEUDOCODE — detect_munqati(narrator_list)Implement and test independently before combining
FUNCTION detect_munqati(narrator_list):

    gaps_found = empty list

    FOR each pair (narrator_A, narrator_B) where B immediately follows A:

        IF either narrator has no tabaqah value:
            skip this pair, record as "generation data unavailable"
            CONTINUE

        gap = absolute_value(narrator_B.tabaqah - narrator_A.tabaqah)

        IF gap is exactly 1:
            record this as a Munqati' gap
            note: which two narrators, gap size = 1
        ELSE IF gap is greater than 1:
            -- Do NOT count this here — it is Mu'dal, handled separately --
            CONTINUE

    RETURN detected = (gaps_found is not empty), details = gaps_found

Mu'dal Detection — Multiple Narrators Missing

PSEUDOCODE — detect_mudal(narrator_list)Similar to Munqati' but gap threshold is 2 or more
FUNCTION detect_mudal(narrator_list):

    severe_gaps = empty list

    FOR each pair (narrator_A, narrator_B) where B immediately follows A:

        IF either has no tabaqah: skip

        gap = absolute_value(narrator_B.tabaqah - narrator_A.tabaqah)

        IF gap >= 2:
            record as Mu'dal gap
            note: which narrators, how many appear to be missing (gap - 1)

    RETURN detected = (severe_gaps is not empty), details = severe_gaps

Mursal Detection — Missing Companion

PSEUDOCODE — detect_mursal(narrator_list)Check chain composition, not consecutive pairs
FUNCTION detect_mursal(narrator_list):

    has_companion = False
    has_successor = False

    FOR EACH narrator in narrator_list:
        IF narrator.tabaqah == 1:
            has_companion = True
        IF narrator.tabaqah == 2:
            has_successor = True

    is_mursal = (has_companion == False) AND (has_successor == True)

    RETURN detected = is_mursal,
           explanation = "Tabi'i narrates without a Companion in chain"
                         if is_mursal else "Companion present in chain"

The Master Analysis Function

PSEUDOCODE — analyze_chain(chain_id)This is what the user interface calls when the user clicks "Analyze"
FUNCTION analyze_chain(chain_id):

    narrator_list = build_chain(chain_id)

    IF narrator_list is empty or error:
        RETURN error to user: "Chain not found"

    results = empty dictionary

    results["munqati"]       = detect_munqati(narrator_list)
    results["mudal"]         = detect_mudal(narrator_list)
    results["mursal"]        = detect_mursal(narrator_list)
    results["mawquf"]        = detect_mawquf(narrator_list)
    results["weak_narrator"] = detect_weak_narrator(narrator_list)

    any_defect_found = check if ANY result has detected = True

    overall_verdict = "Defective Chain" if any_defect_found else "No structural defects detected"

    RETURN narrator_list, results, overall_verdict

    -- Remember: detect_mawquf and detect_weak_narrator pseudocode --
    -- must be written by YOU, using the concepts from Chapter 3.  --
🚫
Academic integrity: The pseudocode above describes the logic. The Python implementation — syntax, data types, method signatures, class structure, error handling, query strings, output formatting — is your work. Do not paste this pseudocode into an AI tool and ask for working Python. Your supervisor will ask you to modify any function live during the viva. If you cannot do it, it will be immediately apparent that you did not write it.
Chapter 16 — Assignment 1

SRS Assignment — Software Requirements Specification

The SRS is your first formal assignment and the foundation of your entire project. It describes WHAT your system will do — from the perspective of the user — without describing HOW it will do it. You should not mention Python, Scikit-learn, Pandas, SQLite, or any technology in the functional requirements section. The SRS template you have been given contains six sections. Each is explained in detail below.

📐
Standard followed: Your SRS follows the IEEE 830 standard, which your VU CS504 course covers in Lectures 3 and 4. Refer to those handouts alongside this guide.

Section 1 — Scope of the Project

Write a comprehensive description of what IsnadGuard does, why it exists, what problem it solves, who will use it, and what it does not do. The scope should be specific enough that a reader can tell exactly what is and is not part of your system. Include the domain background — explain briefly what an Isnad is and why defect detection matters — so that any evaluator understands the context even without prior Islamic studies knowledge. Refer to Chapters 1 through 3 of this guide for the domain content.

Section 2 — Functional and Non-Functional Requirements

This is the most technically important section of your SRS. Functional requirements describe specific behaviors your system must perform. Every requirement must start with "The system shall..." — this is the IEEE 830 standard phrasing. Non-functional requirements describe quality attributes — how well the system performs, how secure it is, how usable it must be.

Examples of correctly written functional requirements for IsnadGuard:

  • The system shall allow the user to load the Hadith narrator chain database from a local file path.
  • The system shall reconstruct a complete narrator chain for any given chain identifier from the loaded database.
  • The system shall detect and report the presence of a Munqati' defect by identifying generation gaps between consecutive narrators.
  • The system shall display each narrator's name, reliability grade, and generation number in the chain view.
  • The system shall train a classification model on extracted chain features and display accuracy, precision, recall, and F1 score on the test set.
  • The system shall provide an explanation for each chain grade prediction showing the contribution of each input feature.

Examples of non-functional requirements:

  • The system shall display defect detection results within three seconds of receiving a chain identifier.
  • The system shall handle chains of up to fifty narrators without performance degradation.
  • The system's interface shall be usable by a user with no prior knowledge of Hadith science — all technical terms shall be accompanied by plain-English explanations.

Section 3 — Use Case Diagram

A Use Case Diagram shows the actors in the system and the actions they can perform. It is a high-level overview — it does not show steps or sequences, only capabilities.

SRS — Section 3
Use Case Diagram

Draw one overall diagram with two actors: User (the student or researcher running the application) and System (the application itself, which is not explicitly drawn but implied). Use cases include: Load Dataset, View Statistics, Search Chain, View Defect Report, View Chain Graph, Classify Chain, View Explanation, Export Report. Connect each actor to the use cases they can initiate.

draw.io — UML → Use Case shapes CS504 reference: Lecture 6 — Use Case Diagrams with worked examples
📌
How to draw this in draw.io: Go to https://app.diagrams.net → New → Blank diagram. In the left panel, search "UML". Drag an Actor shape for each actor and Oval shapes for each use case. Connect them with lines. Use the include/extend relationship arrows only when one use case inherently requires another (for example, "Classify Chain" includes "Load Dataset"). Export as PNG: File → Export As → PNG.

Section 4 — Usage Scenarios

A Usage Scenario is a structured description of one complete use case. The template requires you to write a scenario for every use case shown in the diagram. Each scenario is presented as a table with these fields:

FieldDescriptionExample for "Search Chain"
Use Case TitleThe name of the use caseSearch Chain by Identifier
Use Case IDA unique reference codeUC-03
ActorWho initiates this use caseUser
DescriptionWhat the use case accomplishesThe user enters a chain ID and the system retrieves and displays the full narrator chain with all biographical attributes.
Pre-ConditionsWhat must be true before this use case beginsThe dataset has been successfully loaded (UC-01 is complete).
Actions (Main Path)The normal, successful sequence of steps1. User enters a chain ID in the search box. 2. User clicks Analyze. 3. System retrieves chain from database. 4. System displays narrator list and runs defect detection.
Alternative PathsValid variations of the normal pathUser selects chain from a browsable list instead of typing the ID.
ExceptionsError conditionsChain ID does not exist — system displays "No chain found with this identifier."
Post-ConditionsWhat is true after the use case completesDefect report and chain visualization are displayed on screen.
AuthorYour student IDYour VU student ID

Section 5 — Adopted Methodology

Your project follows the VU Process Model, which combines elements of the Waterfall and Spiral models. Explain this choice specifically for your project: the Waterfall component means you follow sequential phases (requirements → design → implementation → testing → delivery), while the Spiral component means you revisit and refine each phase incrementally — for example, your prototype in Assignment 3 is an early spiral iteration, and your Final Project is the complete spiral. Reference CS605 (SE2) for the formal methodology description. Also mention that the four-assignment structure (SRS, Design Document, Prototype, Final) maps directly to the phases of this combined model.

Section 6 — Work Plan (Gantt Chart)

Create a Gantt chart using Microsoft Project or any free alternative (GanttProject, TeamGantt free tier, or even Excel's bar chart feature). The chart must show all eight modules of your project, organized across two semesters, with the four assignment submission dates marked as milestones. Each module should have a start date, an end date, and its corresponding assignment deliverable. Export the chart as PNG and insert it into your SRS document.

Chapter 17 — Assignment 2

Design Document — How the System Is Built

The Design Document is your second assignment and describes HOW your system works internally. Where the SRS said "the system shall classify chains," the Design Document explains what classes implement this, what database queries are used, how the UI is structured, and what the data flow looks like. The template you have been given contains eight sections.

Section 1 — Introduction to the Design Document

Describe what this document contains and why the Design phase matters in a software development project. Explain that design decisions made here — class structure, database access strategy, module dependencies — prevent costly mistakes during implementation. State which design standards you followed (IEEE 1016 for design documents, which CS504 Lectures 23–25 cover). Briefly list the eight sections of the document and summarize what each one contributes.

Section 2 — Entity Relationship Diagram (ERD)

The ERD shows the structure of your data — the tables in hadith-kg.db, their attributes, and the relationships between them. You must show all five tables: rawis, sanads, sanad_rawis, hadiths, and collections. For each table, show its primary key, all important attributes with their data types, and the foreign key relationships connecting the tables.

Design Document — Section 2
ERD — Database Schema

Show all five tables of hadith-kg.db with their primary keys (id fields), relevant columns, and foreign key relationships. Key relationships: rawis ↔ sanad_rawis via rawi_id; sanads ↔ sanad_rawis via sanad_id; sanads ↔ hadiths via hadith_id; hadiths ↔ collections via collection_id.

dbdiagram.io — free, clean ERDs from text CS403 reference: ERD lectures — entity types, relationships, cardinality notation

Section 3 — Sequence Diagrams

A Sequence Diagram shows the order of messages and method calls between objects during one specific use case. The template requires one sequence diagram per use case from your SRS. For your project, you should draw sequence diagrams for at minimum: loading the dataset, analyzing a chain (defect detection), classifying a chain (ML), and exporting a report.

Design Document — Section 3
Sequence Diagram — Analyze Chain

Objects involved: User, Dashboard (UI), DataLoader, ChainBuilder, DefectDetector, ChainVisualizer. Sequence: User enters chain ID → Dashboard calls ChainBuilder.build_chain(id) → ChainBuilder queries database → returns narrator list → Dashboard calls DefectDetector.analyze(narrator_list) → returns defect report → Dashboard calls ChainVisualizer.draw(narrator_list, defects) → returns graph image → Dashboard displays all results to User.

draw.io — UML → Sequence shapes CS504 reference: Lectures 23–25 — Sequence Diagrams in architectural design

Section 4 — Architecture Design Diagram

The Architecture Design shows how your system is organized into layers or tiers. For IsnadGuard, use a three-tier architecture: a Presentation Layer (the Streamlit dashboard the user sees), a Business Logic Layer (your Python modules — data loader, chain builder, defect detector, ML classifier, XAI explainer, report generator), and a Data Layer (the SQLite database). Show how these layers connect and which direction communication flows.

Design Document — Section 4
Three-Tier Architecture

Layer 1 (Presentation): Streamlit web interface — screens for data loading, chain search, statistics, ML prediction, and report export. Layer 2 (Business Logic): DataLoader, ChainBuilder, DefectDetector, ChainVisualizer, StatisticsEngine, MLClassifier, XAIExplainer, ReportGenerator. Layer 3 (Data): hadith-kg.db SQLite database accessed via Python's sqlite3 library.

draw.io — General → box and arrow layout CS504 reference: Lecture 23 — Architectural Views; Lecture 24–25 — Architectural Models

Section 5 — Class Diagram

The Class Diagram shows the object-oriented structure of your Python code — the classes, their attributes, their methods, and the relationships between them. This is one of the most important diagrams in the Design Document, and your supervisor will use it to verify that you have properly planned your code architecture before implementing it.

Design Document — Section 5
Class Diagram

Classes to include: DataLoader (methods: connect(), load_narrators(), load_chains(), close()); NarratorChain (attributes: sanad_id, narrator_list, length; methods: get_names(), get_grades(), get_generations()); DefectDetector (methods: detect_all(), detect_munqati(), detect_mudal(), detect_mursal(), detect_mawquf(), detect_weak()); ChainVisualizer (methods: draw_chain(), export_image()); MLClassifier (methods: extract_features(), train(), predict(), evaluate()); XAIExplainer (methods: explain(chain, model)); ReportGenerator (methods: generate_pdf()). Show relationships: Dashboard uses all these classes; MLClassifier depends on DataLoader.

draw.io — UML → Class shapes CS304 reference: OOP class diagram lectures — attributes, methods, visibility, relationships

Section 6 — Database Design

This section goes deeper than the ERD — it provides the actual schema of the database you are using, including data types for every column, constraints (NOT NULL, PRIMARY KEY, FOREIGN KEY), and indexes. Since you are using an existing database (hadith-kg.db) rather than designing one from scratch, this section should document the relevant tables' exact schemas as they appear in the database, along with an explanation of how your application reads from each one. You can obtain the exact schema by running PRAGMA table_info(table_name) on each table in your exploration script.

Section 7 — Interface Design

Provide three to four screenshots or hand-drawn mockups of the most important screens in your application. For IsnadGuard, show: the data loading screen (file selector and summary panel), the chain analysis screen (search box, narrator list, defect report, chain graph), the ML and XAI screen (predicted grade, confidence, SHAP chart), and the export/report screen. These do not need to be final — you can update them in your Final Project submission if the actual implementation looks different.

Section 8 — Test Cases

Write test cases for each use case scenario from the SRS. Each test case must have: a test case ID, the use case it tests, pre-conditions, test input, expected output, and pass/fail criteria. For example, for the defect detection function: Input = chain with a known Munqati' gap (you can construct this from your exploration data); Expected Output = detect_munqati returns detected=True with the correct position; Pass Criteria = output matches expected and no exception is raised.

Chapter 18 — Assignment 3

Prototype Assignment — What to Build and How to Submit

The Prototype assignment covers 30–40% of your total project work and corresponds to Modules 1 through 4. At the end of Semester 1, you must submit a working Python application that demonstrates the core data loading, chain construction, defect detection, and visualization pipeline using real hadith-kg data.

What Must Be Working in the Prototype

  • Module 1: Application successfully connects to hadith-kg.db, loads narrator and chain data, displays a summary panel with total counts and grade distribution, and reports any records with missing data.
  • Module 2: Given any valid chain ID, the application reconstructs the ordered narrator list with all required attributes.
  • Module 3: The DefectDetector correctly identifies Munqati', Mu'dal, Mursal, Mawquf, and weak narrator defects. Test it on at least 10 different chains — 5 with known defects and 5 that are structurally sound.
  • Module 4: The chain visualization renders a colored graph showing narrator grades and marking any broken links. Export works and produces a readable PNG image.

What You Submit

  • A ZIP file (maximum 25 MB) containing all Python source files
  • A README.txt file with step-by-step instructions to run your application — including which Python version, which libraries to install (with version numbers), and the exact command to start the application
  • Do not include hadith-kg.db in the ZIP — it is too large. Your README must include instructions for downloading it
  • Screenshots of each module working correctly — at least one screenshot per module
  • Upload to VULMS under the Prototype Assignment submission
🎓
OOP concepts to demonstrate in your prototype: Your code must use classes and objects (not just standalone functions), inheritance or association between classes, encapsulation (keeping each module's logic inside its own class), and exception handling throughout. These are the concepts your supervisor will check during the prototype viva.
Chapter 19 — Assignment 4

Final Deliverable — The Complete System

The Final Project submission is the complete, integrated, tested IsnadGuard system covering all eight modules. It must run as a single Streamlit application that a user can open, load the database, and use to analyze any chain in the dataset. Every module must be functional, tested, and connected to the others through the shared data layer.

Minimum Requirements for Full Marks

  • All eight modules are complete and integrated into the Streamlit dashboard
  • The ML classifier achieves at least 70% accuracy on the test set (20% holdout from hadith-kg)
  • SHAP explanations are displayed for every chain classification
  • The statistics dashboard (Module 5) shows at least four meaningful charts about the dataset
  • PDF report generation works and produces a clean, readable document
  • All code is in Python, organized into classes with proper docstrings and exception handling
  • The application runs from a README.txt with a single command

What to Submit

  • ZIP file with all Python source files, organized into a package structure
  • A requirements.txt file listing every library with version numbers
  • README.txt with complete installation and run instructions
  • A sample PDF report generated by your application from a real chain
  • Screenshots of every screen in the application
Chapter 20 — Project Overview

All Eight Modules — The Complete Picture

1
Data Loading and Preprocessing

Connects to hadith-kg.db, loads and cleans narrator and chain data, standardizes grade labels, handles missing values, displays a summary panel. Output: clean, queryable data representation and statistics.

Semester 1 — Prototype
2
Chain Construction and Representation

Reconstructs individual narrator chains from the database as ordered structures with all required biographical attributes. Output: NarratorChain objects ready for detection and visualization.

Semester 1 — Prototype
3
Rule-Based Defect Detection

Implements all five defect detection functions. Produces a structured defect report for any chain. Output: detailed report identifying defect type, position, and plain-English explanation.

Semester 1 — Prototype
4
Chain Visualization

Renders narrator chains as directed graph diagrams using NetworkX, colored by grade, with broken links highlighted. Output: exportable graph image for reports and on-screen display.

Semester 1 — Prototype
5
Statistical Analysis Dashboard

Presents aggregate statistics — defect type frequencies, grade distributions, most-defective collections, chain length patterns — as charts and summary tables.

Semester 2 — Final
6
Machine Learning Classifier

Extracts features from chains, trains and compares three classifiers (Decision Tree, Random Forest, Logistic Regression), evaluates with accuracy, F1, and confusion matrix. Output: trained model and evaluation report.

Semester 2 — Final
7
Explainability Layer (XAI)

Applies SHAP to the trained classifier to explain each individual prediction, showing which features contributed most and linking them back to specific defect types.

Semester 2 — Final
8
Interactive Dashboard and Report Export

Integrates all modules into a Streamlit web application. User enters any chain ID and gets full defect report, chain graph, ML grade, and XAI explanation in one view. One-click PDF export.

Semester 2 — Final
Chapter 21 — Your Growth

What You Will Learn From This Project

Most final year projects teach students one or two technical skills. This project covers the full spectrum of modern data science and AI engineering — from database querying through machine learning to explainability to deployment. Every module you complete adds a skill that is genuinely in demand, both in Pakistan's growing technology sector and in graduate-level academic work internationally.

SQLite and Relational Databases
Writing real JOIN queries, handling NULL values, and using aggregate functions on a genuine research database — without any abstraction layer hiding the SQL.
Data Cleaning and Wrangling
The skill that takes 60–70% of any real data science project's time. Handling inconsistent labels, missing values, and encoding problems in government-scale data.
Graph Data Structures
Representing and analyzing networks using NetworkX — a skill used in fraud detection, social network analysis, knowledge graphs, and recommendation systems.
Rule-Based System Design
Building auditable detection logic from domain knowledge — a critical skill in medical informatics, legal technology, and compliance systems.
Feature Engineering
Transforming raw domain objects — narrator chains — into meaningful numerical representations. Often the single most important determinant of ML model quality.
Supervised ML Classification
Training, comparing, and evaluating multiple classifiers using Scikit-learn. Understanding the confusion matrix and knowing when accuracy is a misleading metric.
Explainable AI (SHAP)
A skill in very high demand as AI is used in high-stakes decisions. The ability to explain why a model predicted what it did is increasingly a legal and ethical requirement.
Streamlit App Development
Building interactive web dashboards entirely in Python without HTML or CSS. The standard way data scientists present their work to non-technical stakeholders.
Software Engineering Practice
Writing an SRS and Design Document to IEEE standards. A professional skill that distinguishes engineers who can plan from those who can only code.
Digital Islamic Humanities
A rare interdisciplinary competency. Experience applying CS methods to Islamic textual traditions distinguishes your academic profile for graduate study and research positions.

The Bigger Picture

The dataset you are working with has been referenced in multiple peer-reviewed publications on Hadith narrator analysis. The specific combination your project implements — rule-based detection plus supervised ML plus SHAP explainability — has not been published as a complete system at the time this guide was written. If you complete all eight modules seriously and write your work up carefully, this project has genuine potential as a conference paper. Your supervisor can advise on publication pathways once the project is complete.

Chapter 22 — Self-Assessment

Self-Test Questions — Verify Your Understanding

The ten questions below are the type your supervisor is most likely to ask during the viva. Try each one in your own words before revealing the answer. If you can answer all ten fluently without referring to any notes, you are genuinely prepared for the viva. If there are questions you cannot answer, return to the relevant chapter before your viva date.

What is the difference between the Isnad and the Matn of a Hadith, and which one does your project work with?▾
The Isnad is the chain of narrators who transmitted the Hadith — an ordered sequence of people reaching back to the Prophet ﷺ. The Matn is the actual content — what was said or done. IsnadGuard works entirely with the Isnad. It analyzes the structural properties of the narrator chain — generation numbers, reliability grades, and connection patterns — to detect defects. The Matn is present in the database but is not used in your analysis.
What is the difference between Munqati' and Mu'dal, and how does the detection logic differ between them?▾
Both defects involve missing narrators in the chain. Munqati' has exactly one narrator missing — you detect it by finding a generation gap of exactly 1 between two consecutive narrators. Mu'dal has two or more narrators missing consecutively — you detect it by finding a generation gap of 2 or more. They are separate functions in your code and reported separately in the defect report, because they have different implications for the chain's grade.
Which two database tables must you join to reconstruct a narrator chain, and what column connects them?▾
You join the sanad_rawis table with the rawis table. The connecting column is rawi_id in sanad_rawis, which is a foreign key referencing the id column in rawis. You filter sanad_rawis by sanad_id to get all narrator positions for one chain, join to get the biographical data from rawis, and order by the pos column to reconstruct the correct sequence.
Why do you use Random Forest instead of a single Decision Tree for the final ML model?▾
A single Decision Tree tends to overfit — it learns the training data too precisely and performs poorly on new data. Random Forest builds hundreds of trees, each trained on a slightly different random subset of the training data, and takes a majority vote. This averaging process drastically reduces overfitting, produces higher accuracy on the test set, and also generates feature importance scores, which are useful for interpreting which chain features matter most for classification.
What is SHAP, what does it output for a single prediction, and why is it especially important in this specific domain?▾
SHAP (SHapley Additive exPlanations) is a method that explains individual machine learning predictions by assigning each input feature a contribution score — called a Shapley value — that represents how much it pushed the prediction toward or away from each class. For a single chain prediction, SHAP outputs a bar chart showing, for example, that max_gen_gap contributed +0.42 toward Da'if while chain_length contributed -0.08. In Hadith authentication specifically, SHAP is important because the tradition requires transparency of reasoning — a scholar cannot accept a Da'if verdict without knowing which part of the chain caused it. An unexplained black-box result has no scholarly value.
What is feature engineering and what seven features have you designed for the ML classifier?▾
Feature engineering is the process of converting raw data into meaningful numerical representations that a machine learning model can learn from. The seven features for this project are: chain_length (number of narrators), max_gen_gap (largest generation gap between consecutive narrators), total_gen_gaps (count of all gaps above 1), weak_narrator_count (narrators with Da'if or worse grade), min_grade_score (minimum numerical grade across the chain), has_companion (binary: 1 if Gen 1 narrator present), ends_at_companion (binary: 1 if last narrator is Gen 1). All seven are computed from the tabaqah and grade columns in the rawis table.
What is the three-tier architecture of your system and what belongs in each tier?▾
The Presentation Layer is the Streamlit web interface — all the screens and controls that the user interacts with. The Business Logic Layer contains all the Python classes that do the actual work: DataLoader, ChainBuilder, DefectDetector, ChainVisualizer, StatisticsEngine, MLClassifier, XAIExplainer, and ReportGenerator. The Data Layer is the SQLite database — hadith-kg.db — which is accessed by the DataLoader class via Python's built-in sqlite3 library. The Presentation Layer communicates only with the Business Logic Layer; only the Business Logic Layer communicates directly with the Data Layer.
Your supervisor asks you to run a chain that you have never tested before. Walk through what happens in your system step by step.▾
The user enters the chain ID in the search box on the Streamlit dashboard and clicks Analyze. The Dashboard calls ChainBuilder.build_chain(id), which queries sanad_rawis joined with rawis filtered by that sanad_id and ordered by pos. The result is a list of narrator objects. The Dashboard calls DefectDetector.analyze_chain(narrator_list), which runs all five detection functions and returns a results dictionary. The Dashboard calls ChainVisualizer.draw_chain(narrator_list, results) to generate the graph image. All results — narrator list, defect report, graph — are displayed simultaneously on screen. The user can then click Classify Chain to trigger the ML prediction and XAI explanation.
What is the difference between accuracy and recall as evaluation metrics, and which is more important for your project?▾
Accuracy is the percentage of all predictions that are correct. Recall (for the Da'if class) is the percentage of truly Da'if chains that your model correctly identifies. In your project, recall is more important than accuracy. Consider this: if your dataset has 80% Sahih chains and 20% Da'if chains, a model that always predicts Sahih gets 80% accuracy while catching 0% of the defective chains. That model is useless for the actual purpose of the system. High recall on the Da'if class means the system catches most of the actually defective chains — which is the real goal.
Your connection dropped during the Python download. What exactly do you do next?▾
Nothing special. Simply re-run the same download_dataset.py script — the command is python download_dataset.py. The Hugging Face library automatically detects the .incomplete partial file saved in the ./data/ folder and resumes from the byte position where the download stopped. You do not need to delete any files or change any parameters. The script will print confirmation when the download completes and report the file size, which you should verify is approximately 1.6 GB.