ia-framework

Framework Redesign: What I Learned

A comprehensive audit revealed systemic architectural problems in the IA framework. Here's what patterns worked consistently, what caused failures, and why a complete rebuild is the only viable path forward.

Chris Groves

Chris Groves

— 6 min read
Hero image for Framework Redesign: What We Learned from Breaking and Rebuilding Intelligence Adjacent

The Breaking Point

After months of building the Intelligence Adjacent framework, certain workflows produced exceptional results. Security assessments, compliance evaluations, wellness programs, and research tasks all delivered consistently excellent output when they followed proper structural patterns. The architecture worked brilliantly—but only when used correctly.

Other workflows failed unpredictably. Agents couldn't find code they needed. Routing decisions varied between identical requests. Validation hooks caught architectural violations that shouldn't have been possible. The framework worked brilliantly—until it didn't.

The pattern became impossible to ignore: Success correlated with specific structural choices. Failure correlated with their absence.

My comprehensive audit revealed the architectural mismatch wasn't fixable through incremental improvements. The foundation needed rebuilding.

What Was Working: The 99% Success Pattern

When workflows succeeded, they shared distinctive characteristics:

Co-located Resources: All necessary components lived together in one folder hierarchy. The skill definition, executable code, workflow phases, templates, and reference materials existed as a cohesive unit. Agents loaded skill context and immediately had access to everything required.

Clear Execution Paths: The command invoked an agent. The agent loaded a skill folder. The skill contained integrated code and workflows. No searching across directories. No interpretation of scattered resources. The path from user intent to execution remained deterministic.

Self-Contained Logic: Workflows that succeeded could be understood independently. Reading the skill folder provided the complete picture—no dependencies on framework-level documentation to understand business logic. No routing decisions buried in markdown files.

When developers followed this pattern and proper methodology, output quality remained consistently excellent. The architecture supported reliability.

What Was Broken: The Failure Pattern

Failures revealed systematic architectural problems:

Code Scattering: Executable code dispersed across framework-level directories instead of co-locating with skill definitions. Agents had to search multiple locations to find necessary resources. When code lived separately from the workflows that needed it, discovery became non-deterministic.

Prompt-Based Logic: Critical routing and validation implemented through AI interpretation of markdown documentation rather than deterministic code. Business logic embedded in documentation files. Agents inferred intent from prose descriptions. Small changes in phrasing altered behavior unpredictably.

Multi-Layer Enforcement Needed: The framework required multiple validation hooks to catch edge cases that deterministic architecture would have prevented. Pre-commit hooks checking for violations. Startup hooks validating structure. Runtime hooks enforcing standards. Each layer compensated for non-deterministic prompt-based decisions.

Reference Confusion: Multiple files claiming to be authoritative for the same metadata. Skills defined in one location, referenced in another, with routing logic in a third. No single source of truth. Updates required coordinated changes across scattered files.

Inconsistent Discovery: Some workflows integrated all resources. Others required agents to find code in framework-level directories. The discovery process varied by skill. Agents couldn't reliably locate everything needed for execution.

According to research on separation of concerns, "the application of the Separation of Concerns involves two processes: reduction of coupling and increasing cohesion." The IA framework violated both principles—tight coupling between skills and framework infrastructure, scattered cohesion across directory boundaries.

Root Causes: Architecture Mismatch

My audit identified core architectural problems:

1. No Clear Separation Between Framework and Workflows

The framework mixed two distinct concerns:

  • Infrastructure (how skills load, how agents route, how validation works)
  • Complete workflows (A→Z capabilities with integrated code)

Without clear boundaries, skills depended on framework internals. Changes to infrastructure risked breaking workflows. Workflows couldn't be extracted or understood independently.

2. Non-Deterministic Routing

Microsoft's AI Decision Framework notes that "when the appropriate agents and their order are always known upfront, task routing is simple and deterministically rule-based, not based on dynamic context window or dynamic interpretation."

The IA framework embedded routing logic in prose documentation. Agents interpreted descriptions like "security ASSESSMENT goes to security agent" and "IMPLEMENTATION goes to engineer agent." Small phrasing changes altered routing. Testing routing decisions required testing AI interpretation—an inherently unstable foundation.

3. Resource Co-Location Inconsistency

Some skills followed the co-location principle. All resources in one folder hierarchy. Complete, self-contained capabilities.

Others scattered resources across framework directories. Code in tools/. Workflows in skills/. Templates in library/. Each fragment useless without the others.

Research on modular architecture emphasizes that "modularity is a way of organizing code into smaller, self-contained modules based on logical divisions within application functionality." Scattering resources violated this fundamental principle.

4. Tight Coupling Prevents Modularity

Skills depended on:

  • Agent routing logic in agents/ directory
  • Command definitions in commands/ directory
  • Shared templates in library/ directory
  • Validation hooks in hooks/ directory

Extracting a skill required understanding framework infrastructure. Modifying infrastructure risked breaking skills. The coupling prevented independent evolution.

Carnegie Mellon research found that "when debt is built into the software architecture, it becomes a deep-seated issue challenging to solve or manage without significant investment and time." The IA framework's tight coupling represented architectural debt.

Technical Audit Results

My comprehensive audit measured the scope across the skill system, tool distribution, command routing structure, and validation hooks—recently rebuilt in Python for improved reliability.

Clear Pattern: Co-located code with skill definitions = consistency. Scattered resources = unpredictable behavior.

I avoided claiming certainty about individual components. The pattern mattered more than specific counts. Where resources lived together, reliability followed. Where they scattered, problems emerged.

Lessons Learned: Principles for V2

1. Measure Before Changing

I assessed what actually worked before proposing fixes. The success pattern proved more instructive than the failures. Understanding why certain workflows succeeded provided the blueprint for architectural improvements.

2. Code Over Prompts for Critical Paths

AI agent routing research shows that "deterministic chains work well for well-defined tasks with predictable workflows, when consistency and auditing are top priorities, and when you want to minimize latency by avoiding multiple LLM calls for orchestration decisions."

Use deterministic code for:

  • Routing decisions (command → agent → skill)
  • Validation rules (structure, credentials, standards)
  • Workflow state management (phase transitions)

Use AI interpretation for:

  • Content generation
  • Natural language understanding
  • Creative problem-solving
  • Research synthesis

3. The Co-Location Principle

Resources needed together should live together. Keep skill code, workflows, templates, and reference materials in one folder hierarchy. Enable agents to load skill context and immediately access everything required.

Modular programming research confirms: "organize your codebase into directories and files that reflect the modular structure of your application." Co-location supports comprehension and reliability.

4. Single Source of Truth

Metadata should have one authoritative location. Not scattered across multiple files. Not duplicated with synchronization requirements. One definition. All references point to it.

5. Enforce by Structure, Not Hooks

Make architectural mistakes impossible through code design rather than relying on validation to catch them. Structure the codebase so wrong paths don't exist. Hooks should validate external inputs, not compensate for architectural fragility.

6. Optimize for Agent Discovery

The easier it is for agents to find all resources for a workflow, the more reliable the results. Reduce cognitive load. Minimize search patterns. Provide clear, predictable resource organization.

Research on AI agent architecture patterns emphasizes that "large language models (LLM) based approaches are the state-of-the-art for agent routing, relying on an LLM's pre-trained knowledge, coupled with prompt engineering techniques, to route a user query to the relevant agent." The architecture should support AI discovery patterns, not fight them.

Why Rebuild vs Patch

The tight coupling prevents incremental fixes. Changing one element cascades through the system:

  • Moving code from tools/ to skills/ requires updating agent prompts
  • Updating agent routing requires changing command definitions
  • Modifying skill structure requires rewriting validation hooks
  • Each change risks breaking working workflows

Research on technical debt management confirms this reality. Studies show that "poor practices for dealing with tech debt correlate with up to 40% higher likelihood of canceled or unfinished modernization projects." Attempting to patch architectural debt often fails.

A complete rebuild in a new location provides:

Clean Foundation: Apply lessons learned from the start. Structure for success.

Progressive Migration: Move working patterns to new architecture. Validate improvements.

No Contamination: Avoid inheriting scattered patterns or legacy assumptions.

Comparison Capability: Maintain old system during transition. Validate new approach before decommissioning.

Clear Before/After: Document old problems and new solutions with concrete examples.

The rebuild strategy follows AWS modernization guidance: "the goal isn't to eliminate technical debt entirely but to manage it strategically while maintaining business agility."

What This Means: Forward Path

This isn't about failure. The framework achieved its goals across different domains. Successful workflows proved the concept. The architecture discovered better patterns through real-world use.

The audit revealed that specific structural decisions created fragility:

  • Scattering code reduced reliability
  • Non-deterministic routing required excessive enforcement layers
  • Tight coupling prevented modularity
  • Multiple sources of truth caused reference confusion

A rebuild using these lessons will create a more robust, maintainable system:

Features (core/): Framework infrastructure—agents, skills system, workflow engine, enforcement, prompting. The machinery that makes workflows possible.

Skills (skills/): Complete A→Z capabilities with co-located code, workflows, reference materials, and templates. Self-contained, extractable, independently comprehensible.

Deterministic routing, validation, and workflow management using code instead of prompts. Predictable behavior. Auditable decisions.

Modular skills where all necessary resources exist in the same folder hierarchy. Clear boundaries. Minimal coupling.

The path forward builds on proven patterns. Co-location worked—expand it. Deterministic routing worked—enforce it. Self-contained workflows worked—make them the standard.

Research on modular software development confirms this direction: "testing becomes more straightforward with modular code, as each module can be tested independently, allowing you to identify and fix issues quickly."

The redesign implements lessons learned through building, measuring, and discovering what actually works.

Sources

Read more