PROBLEM
Agencies possess exceptionally large and unique datasets that can yield new, critical national security insights with the help of generative AI. Intelligence officers, however, frequently perform time-consuming manual tasks that could be off-loaded to AI-enabled tools. Security requirements and other significant barriers hinder government experimentation with commercially-available LLMs.
ANSWER
A customized 3rd-party benchmark that scores models for zero-shot performance against common intelligence officer use cases will ensure limited resources are allocated to further testing/evaluation of the most promising capabilities.
AUDIENCE
Seeking independent evaluations of AI tool efficacy, at no risk to government systems or data.
- Validation for companies already marketing products to government.
- Valuable training and development feedback to new product innovators.
- Advisory/custom benchmarking services for govt and providers seeking LLM fine-tuning support.
- Product vetting for the dual use investor community.