Harness Engineering Is Just-in-Time In-Context Learning

While the generally capable models will always become more capable, there’s nothing that requires the models’ priors around what good looks like to align with your own. It follows that it will always be required to curate the environment around the model such that it spikes in the direction of coherent choices for the nonfunctional requirements you or your organization will accept as good work.

No amount of making the model better will obsolete this need for in-context learning (ICL).

All of “harness engineering” is essentially a set of tricks to provide JIT opportunities for ICL to align model behavior with what good looks like for you without unduly restraining these reasoning models.