Query-Centric Keyphrase Scoring

The problem

Standard search results often fail to prioritize the specific core concepts of a user's question. It is difficult to find the most relevant information when the system treats every word in a query with equal importance.

What it does

The tool identifies the most important concepts in a search query and reorders a list of text snippets based on how often those specific concepts appear. It ranks the results by focusing on the core meaning of the user's request.

Why it matters

It ensures that the most relevant information is surfaced by prioritizing the specific topics a user actually cares about.

Validation

It was run in the sandbox and it failed. run output shows an error/traceback — the artifact does NOT run clean.

$ python3 query_centric_reranker.py
Traceback (most recent call last):
  File "/work/query_centric_reranker.py", line 42, in <module>
    reranker = QueryCentricReranker(query)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/query_centric_reranker.py", line 9, in __init__
    self.key_terms = self._extract_key_terms(query, num_key_terms)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/query_centric_reranker.py", line 17, in _extract_key_terms
    filtered = [token for token in tokens if token not in self.stop_words]
                                                          ^^^^^^^^^^^^^^^
AttributeError: 'QueryCentricReranker' object has no attribute 'stop_words'

No screenshot — there is nothing working to show. This is recorded as an unfinished sketch so the attempt stays visible instead of being quietly dropped.

The code

All of it — 62 lines, one file, standard library only.

import re
from collections import Counter
import string

class QueryCentricReranker:
    def __init__(self, query, num_key_terms=10):
        self.key_terms = self._extract_key_terms(query, num_key_terms)
        try:
            from nltk.corpus import stopwords
            self.stop_words = set(stopwords.words('english'))
        except:
            self.stop_words = set(string.ascii_lowercase)

    def _extract_key_terms(self, query, num_terms):
        tokens = re.findall(r'\b\w+\b', query.lower())
        filtered = [t for t in tokens if t not in self.stop_words]
        return [word for word, _ in Counter(filtered).most_common(num_terms)]

    def score_document(self, document):
        words = re.findall(r'\b\w+\b', document.lower())
        term_weight_sum = sum(self._get_term_weight(word) for word in words)
        return term_weight_sum / len(words) if words else 0

    def _get_term_weight(self, term):
        return 1.5 if term in self.key_terms else 0.5

    def rerank(self, documents):
        # Calculate document frequency for key terms
        doc_freq = {term: 0 for term in self.key_terms}
        for doc in documents:
            words = re.findall(r'\b\w+\b', doc.lower())
            for term in self.key_terms:
                if term in words:
                    doc_freq[term] += 1
                    break

        # Calculate weights (inverse document frequency)
        total_docs = len(documents)
        self.term_weights = {term: (total_docs - df + 1) / (df + 1) for term, df in doc_freq.items()}

        # Score documents with weighted terms
        scored = []
        for doc in documents:
            words = re.findall(r'\b\w+\b', doc.lower())
            term_weights = sum(self.term_weights.get(word, 0.5) for word in words)
            score = term_weights / len(words) if words else 0
            scored.append((doc, score))

        return [doc for doc, _ in sorted(scored, key=lambda x: x[1], reverse=True)]

if __name__ == "__main__":
    query = "machine learning algorithms"
    documents = [
        "This paper discusses machine learning algorithms."
        "Algorithms are used in various fields."
        "Machine learning is a subset of AI."
    ]
    reranker = QueryCentricReranker(query)
    ranked_docs = reranker.rerank(documents)
    print("Reranked documents:\n")
    for doc in ranked_docs:
        print(f"- {doc}")