Pre-Ingestion Epistemic Filter

The problem

It is difficult to know which parts of a piece of information are reliable or uncertain before you actually process it. This makes it hard to separate facts from guesswork.

What it does

It scans text to identify missing markers of uncertainty and then scores those points based on a flow-control system. It essentially flags how much doubt should be attached to specific pieces of data.

Why it matters

It allows for a more accurate assessment of information reliability before it is integrated into a system.

Validation

It was run inside an isolated container with no network access. This is the exact command and the real output it produced — captured process output, not written by a model.

$ python3 pre_ingestion_filter.py
File "/work/filter.py", line 90
    input_text = file.read()
IndentationError: unexpected indent

A screenshot of that run.

A clean run proves this does what is shown above, in a CPU-only sandbox. It is a small research demo — not a production tool, and nothing here was published anywhere.

The code

All of it — 86 lines, one file, standard library only.

import re
from datetime import datetime, timedelta

class TokenBucket:
    """Token bucket rate-limiter"""
    def __init__(self, capacity, fill_rate):
        self.capacity = capacity
        self.fill_rate = fill_rate
        self.tokens = capacity
        self.last_filled = datetime.now()
        self.request_history = []

    def allow_request(self, cost=1):
        now = datetime.now()
        time_passed = (now - self.last_filled).total_seconds()
        new_tokens = time_passed * self.fill_rate / 60  # tokens per minute
        self.tokens = min(self.capacity, self.tokens + new_tokens)
        self.last_filled = now
        
        if self.tokens >= cost:
            self.tokens -= cost
            self.request_history.append((now, cost))
            return True
        return False

class ClarityGate:
    """Identifies uncertainty markers"""
    def __init__(self):
        self.uncertainty_markers = re.compile(r'(speculation|hypothesis|assumption|uncertain|possibly|maybe|likely|estimates|approximately|projection|suggest)', re.IGNORECASE)

    def check_for_uncertainty(self, text):
        matches = self.uncertainty_markers.findall(text)
        return len(matches) > 0

    def score_clarity(self, text):
        """Score based on uncertainty markers"""
        total = len(text)
        uncertainty_count = len(self.uncertainty_markers.findall(text))
        return uncertainty_count / total if total > 0 else 0

class EpistemicFilter:
    """Combined filter"""
    def __init__(self, bucket_capacity=5, bucket_fill_rate=1):
        self.bucket = TokenBucket(bucket_capacity, bucket_fill_rate)
        self.clarity_gate = ClarityGate()

    def process_text(self, text):
        # Rate limiting check
        if not self.bucket.allow_request(cost=len(text)/100):
            # Cost based on text length
            return {
                'allowed': False,
                'reason': 'Rate limit exceeded',
                'clarity_score': 0,
                'epistemic_density_score': 0
            }

        # Uncertainty marking check
        matches = self.clarity_gate.uncertainty_markers.findall(text)
        uncertainty_count = len(matches)
        words = re.findall(r'\w+', text)
        word_count = len(words)
        
        uncertainty_present = len(matches) > 0
        clarity_score = uncertainty_count / len(text) if len(text) > 0 else 0
        epistemic_density_score = uncertainty_count / word_count if word_count > 0 else 0
        
        return {
            'allowed': True,
            'clarity_score': clarity_score,
            'uncertainty_present': uncertainty_present,
            'token_balance': self.bucket.tokens,
            'epistemic_density_score': epistemic_density_score
        }

# Example usage
if __name__ == "__main__":
    filter = EpistemicFilter()
    test_text = """
    This is a text that contains speculation about future events. 
    It may or may not happen, but our hypothesis suggests it's likely.
    """
    result = filter.process_text(test_text)
    print(f"Processing result: {result}")
    print(f"Token bucket status: {filter.bucket.tokens} tokens remaining")
    print(f"Epistemic Density Score: {result['epistemic_density_score']}")