Falsification-Robust-Trace-Diffing

The problem

It is difficult to identify exactly where a multi-agent system goes wrong when multiple actors are performing complex actions. Tracking these errors is hard because the path taken can be messy and inconsistent.

What it does

It analyzes the steps taken by multiple agents to identify actions that are logically impossible or invalid. It flags these errors by comparing the actual path taken against what a correct sequence should look like.

Why it matters

It provides a clear way to pinpoint exactly where a multi-agent system breaks by identifying invalid steps in its behavior.

Validation

It was run inside an isolated container with no network access. This is the exact command and the real output it produced — captured process output, not written by a model.

$ python3 falsification_robust_trace_diff.py
usage: falsification_robust_trace_diff.py [-h] trace_files trace_files
falsification_robust_trace_diff.py: error: the following arguments are required: trace_files

A screenshot of that run.

A clean run proves this does what is shown above, in a CPU-only sandbox. It is a small research demo — not a production tool, and nothing here was published anywhere.

The code

All of it — 68 lines, one file, standard library only.

# Falsification-Robust Trace Differ (FRTD)

import json
import argparse
from difflib import Differ
from collections import defaultdict

class StateTransitionValidator:
    def __init__(self, trace):
        self.transitions = self._parse_trace(trace)
        self.allowed_transitions = defaultdict(set)
        # Example allowed transitions (to be customized)
        self.allowed_transitions['initial'].add('processing')
        self.allowed_transitions['processing'].add('completed')
        self.allowed_transitions['processing'].add('error')

    def _parse_trace(self, trace_data):
        # Implement trace parsing logic based on your format
        # Returns list of state transitions
        pass

    def validate_transitions(self):
        invalid_transitions = []
        current_state = 'initial'
        for transition in self.transitions:
            next_state = transition.get('next_state')
            if next_state not in self.allowed_transitions[current_state]:
                invalid_transitions.append(transition)
            current_state = next_state
        return invalid_transitions

def compare_traces(trace1, trace2):
    # Compare two traces using difflib
    diff = Differ()
    diff_result = list(diff.compare(trace1, trace2))
    return [line for line in diff_result if line.startswith('+') or line.startswith('-')]

def main():
    parser = argparse.ArgumentParser(description='Falsification-Robust Trace Differ')
    parser.add_argument('trace_files', nargs=2, help='Two trace files to compare')
    args = parser.parse_args()

    with open(args.trace_files[0], 'r') as f1, open(args.trace_files[1], 'r') as f2:
        trace1 = json.load(f1)
        trace2 = json.load(f2)

    validator1 = StateTransitionValidator(trace1)
    validator2 = StateTransitionValidator(trace2)

    invalid1 = validator1.validate_transitions()
    invalid2 = validator2.validate_transitions()

    diff = compare_traces(trace1, trace2)

    print('Invalid transitions in trace 1:')
    for t in invalid1: print(json.dumps(t))
    print('\nInvalid transitions in trace 2:')
    for t in invalid2: print(json.dumps(t))
    print('\nDifferences between traces:')
    for line in diff: print(line)

if __name__ == '__main__':
    main()

# Example usage: 
# python frtd.py trace1.json trace2.json
# Requires two JSON trace files with state transition data
# Outputs invalid transitions and differences between traces