Dynamic Token Budgeting

The problem

AI models can get stuck exploring endless, complex reasoning paths that waste resources and lead to dead ends. It is difficult to balance deep thinking with staying within a set limit.

What it does

It scores different paths of thought and prunes the ones that exceed a set 'budget' of tokens. This ensures the system only explores the most promising directions.

Why it matters

It allows for deep reasoning while keeping the process efficient and controlled.

Validation

It was run inside an isolated container with no network access. This is the exact command and the real output it produced — captured process output, not written by a model.

$ python3 dynamic_token_budgeting.py
Valid paths after token-based pruning:(
)
Path 1: {[node.value for node in path]}

A screenshot of that run.

A clean run proves this does what is shown above, in a CPU-only sandbox. It is a small research demo — not a production tool, and nothing here was published anywhere.

The code

All of it — 56 lines, one file, standard library only.

# Dynamic Token Budgeting System

import time
from dataclasses import dataclass

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity
        self.refill_rate = refill_rate  # tokens per second
        self.tokens = capacity
        self.last_refill = time.time()

    def consume(self, count):
        now = time.time()
        elapsed = now - self.last_refill
        self.tokens += elapsed * self.refill_rate
        self.tokens = min(self.capacity, self.tokens)
        if self.tokens < count:
            return False
        self.tokens -= count
        self.last_refill = now
        return True

class PathScorer:
    def __init__(self, token_bucket):
        self.token_bucket = token_bucket

    def score_paths(self, paths):
        valid_paths = []
        for path in paths:
            total_cost = sum(node.cost for node in path)
            if self.token_bucket.consume(total_cost):
                valid_paths.append(path)
        return valid_paths

@dataclass
class Node:
    value: str
    cost: int

if __name__ == "__main__":
    # Initialize token bucket with 5 tokens, refilling at 1 token/second
    bucket = TokenBucket(capacity=5, refill_rate=1)
    scorer = PathScorer(bucket)

    # Example paths with nodes containing token costs
    paths = [
        [Node("Thought 1", 2), Node("Thought 1a", 1)],
        [Node("Thought 2", 3)],
        [Node("Thought 3", 1), Node("Thought 3a", 1), Node("Thought 3b", 1)]
    ]

    valid_paths = scorer.score_paths(paths)
    print("Valid paths after token-based pruning:(\n)")
    for i, path in enumerate(valid_paths, 1):
        print(f"Path {i}: {{[node.value for node in path]}}")