My work at General Magick Industries
I re-engineered a multi-model LLM reasoning service that explores and combines responses from multiple foundation models to answer complex questions. By optimizing its Monte Carlo Tree Search (MCTS) pipeline through parallelized evaluation and aggregation, I achieved ~5.3× faster execution than the previous implementation.
My responsibilities
- Reasoning implementation and optimization. I owned the MCTS pipeline for candidate generation, evaluation, and iterative refinement, parallelizing evaluation and aggregation to achieve ~5.3× faster execution than the previous implementation.
- Benchmarking and strategy evaluation. I benchmarked LLM reasoning strategies using datasets such as GPQA, assessing answer quality, latency, and inference cost to inform model routing and selection. Successful configurations completed in ~36 seconds.
MCTS-based reasoning and multi-model aggregation
I structured the reasoning process around candidate generation, evaluation, and refinement. MCTS guided which paths to explore further, while aggregation allowed responses from multiple models to contribute to another candidate answer.
- Generate candidates. Concurrent model calls produced alternative responses to the same request.
- Evaluate and refine. Model-based assessments guided further exploration, with evaluator feedback informing subsequent attempts.
- Aggregate and assess. Responses could be combined into a new candidate and evaluated alongside other candidates before final selection.
The engineering tradeoff was how much additional exploration to allow, given the model calls required for both generation and evaluation. I optimized the pipeline through parallelized evaluation and aggregation, achieving ~5.3× faster execution than the previous implementation. Asynchronous execution and bounded search coordinated the work and limited search effort; answer quality still required separate evaluation.
Reliability and usage accounting
The evaluator itself could return a malformed response. I implemented validation and limited retries to provide a recovery path while constraining additional model calls.
Usage records included every evaluation attempt, making the cost of retries available for review. Focused tests covered malformed-response handling and accounting across repeated attempts.
Benchmarking
I contributed to internal benchmarking on GPQA, a benchmark of graduate-level science questions. The evaluation checked answers against reference answers and recorded response latency, answer-parsing failures, and reported model usage.
Successful configurations completed in ~36 seconds. I assessed answer quality alongside latency and inference cost to inform model routing and selection.
In broader internal testing, MCTS appeared useful for complex questions requiring extended reasoning. A single model was often preferable for simpler requests where latency and cost were central. No strategy consistently performed best across the limited set of workloads evaluated.
My evaluation findings informed a proposed routing approach for selecting reasoning strategies according to request complexity, latency, and cost.
Agent architecture and platform services
- Open-source Python SDK. I extended MagickMind’s SDK with multi-model LLM reasoning capabilities, enabling developers and internal AI services to access advanced reasoning through a unified platform API.
- Agent memory architecture. I co-designed and implemented semantic, episodic, and procedural memory with consolidation, forgetting, and memory-decay mechanisms, using CoALA as the foundation.
- Agent personas and personality evolution. I co-designed the architecture for representing personality traits and supporting their evolution over time.
- Usage and billing service. I owned implementation of usage metering and billing infrastructure, including usage tracking, wallet-credit management, invoice validation, and duplicate-event handling.