June 9, 2025
When the Recommendation Was Valid
Identifying opportunities is not the same as deciding which opportunities are worth taking.
That distinction became concrete during AI readiness work at Chainlink.
The recommendation was valid. The score would have improved.
I still chose a different solution.
One of the projects I worked on last year involved improving our documentation's AI readiness. Part of that effort included a third-party evaluation framework that scored how effectively AI systems could retrieve and consume documentation content. Improving the score was an explicit goal, and many of the recommendations were genuinely useful.
One recommendation, however, created an interesting tension.
The framework identified an opportunity to improve how AI systems accessed documentation through content negotiation. The underlying problem was real, and implementing the recommendation would likely have improved our score.
The challenge was that the recommendation wasn't an isolated documentation change.
It would have required shifting documentation routes from a static model to a runtime model, introducing conditional behavior, additional caching complexity, debugging overhead, and tooling compatibility concerns across the platform.
From the perspective of the evaluation framework, the recommendation made sense.
From the perspective of the larger system, the answer was less obvious.
The question wasn't whether the recommendation was valid.
The question was whether it was the right tradeoff.
Documentation systems don't exist solely to optimize retrieval scores. They also need to support developer navigation, maintainability, operational simplicity, and long-term scalability. Improving one dimension can easily create costs elsewhere.
After evaluating the tradeoffs, I ultimately pursued a different approach that addressed the underlying retrieval problem without requiring a fundamental restructuring of the platform. We achieved most of the same outcome while preserving the existing architecture.
What stuck with me wasn't the score improvement.
It was realizing that neither the AI system nor the evaluation framework could make the decision for us.
They could identify opportunities and measure outcomes. What they could not do was evaluate competing constraints and determine which implementation path made the most sense within the broader system.
That required judgment.
I suspect this is where many discussions about AI still miss the point.
For a long time, expertise was closely associated with production. The ability to write, build, analyze, explain, or generate the output itself.
As AI reduces the cost of generation, the bottleneck increasingly shifts elsewhere.
Organizations are not struggling to produce recommendations.
They are increasingly struggling to determine which ones to implement, how to implement them, and what tradeoffs are acceptable along the way.
AI can help identify opportunities for improvement.
Organizations still need people who can evaluate those opportunities against operational reality.
AI does not eliminate expertise.
It relocates it.