My work at General Magick Industries
I co-developed a multi-model reasoning service that explores and combines responses from multiple foundation models to answer complex questions. My responsibility was the design and implementation of its MCTS-based reasoning, connecting candidate generation, evaluation, refinement, and multi-model aggregation.
My responsibilities
- Reasoning design and implementation. I owned the Monte Carlo Tree Search (MCTS) implementation and its integration with model execution, evaluation, and aggregation.
- Benchmarking and strategy evaluation. I contributed to GPQA benchmarking and assessed answer quality, latency, and model-call cost with the team to understand where additional reasoning was justified.
MCTS-based reasoning and multi-model aggregation
I structured the reasoning process around candidate generation, evaluation, and refinement. MCTS guided which paths to explore further, while aggregation allowed responses from multiple models to contribute to another candidate answer.
- Generate candidates. Concurrent model calls produced alternative responses to the same request.
- Evaluate and refine. Model-based assessments guided further exploration, with evaluator feedback informing subsequent attempts.
- Aggregate and assess. Responses could be combined into a new candidate and evaluated alongside other candidates before final selection.
The engineering tradeoff was how much additional exploration to allow, given the model calls required for both generation and evaluation. The implementation used asynchronous execution and bounded search to coordinate that work. These controls limited the search effort; answer quality still required separate evaluation.
Reliability and usage accounting
The evaluator itself could return a malformed response. I implemented validation and limited retries to provide a recovery path while constraining additional model calls.
Usage records included every evaluation attempt, making the cost of retries available for review. Focused tests covered malformed-response handling and accounting across repeated attempts.
Benchmarking
I contributed to internal benchmarking on GPQA, a benchmark of graduate-level science questions. The evaluation checked answers against reference answers and recorded response latency, answer-parsing failures, and reported model usage.
In broader internal testing, MCTS appeared useful for complex questions requiring extended reasoning. A single model was often preferable for simpler requests where latency and cost were central. No strategy consistently performed best across the limited set of workloads evaluated.
My evaluation findings informed a proposed routing approach for selecting reasoning strategies according to request complexity, latency, and cost.
Agent architecture and platform services
- Agent memory architecture. I co-designed semantic, episodic, and procedural memory and the memory consolidation pipeline, applying research on cognitive architectures with CoALA as the foundation.
- Agent personas and personality evolution. I co-designed the architecture for representing personality traits and supporting their evolution over time.
- Usage and billing service. I currently own development of usage metering, wallet credits, and invoice validation, including checks for duplicate usage events and repeated credit requests.