AI Evaluation Subject Matter Expert
Foxhole TechnologyCharleston, South Carolina, United States · Posted 2 months agoDescription
Work Arrangement: Hybrid
Clearance: Active Secret w/TS Capability
Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized leader in navigating technology and security challenges, Foxhole delivers mission-focused innovations to answer evolving and complex needs. Our talented employee-owners provide agile, scalable services and solutions that solve operational gaps, operate critical systems, and protect and secure the enterprise – across the organization and around the world.
Foxhole Technology is seeking an AI Evaluation SME to join an existing program. The AI Evaluation SME will support the assessment, testing, validation, and operational evaluation of artificial intelligence, machine learning, automation, analytics, and decision-support capabilities being considered for or integrated into the Navy’s Next Generation CANES environment. This role will help ensure AI-enabled capabilities are mission-relevant, reliable, secure, explainable, measurable, and suitable for deployment within afloat, tactical, disconnected, intermittent, limited-bandwidth, and multi-security-domain environments.
The SME will develop evaluation frameworks, test methods, metrics, datasets, scenarios, risk assessments, and reporting products that help Navy and CACI stakeholders determine whether AI-enabled capabilities improve network operations, cyber defense, system administration, predictive maintenance, anomaly detection, configuration management, mission planning, or other CANES-related functions.
.
KEY RESPONSIBILITIES:
- Serve as a senior technical advisor for AI evaluation, test planning, performance assessment, and operational suitability analysis in support of Next Generation CANES modernization.
- Develop AI evaluation strategies, test plans, measures of effectiveness, measures of performance, success criteria, risk indicators, and evaluation scorecards.
- Assess AI, machine learning, generative AI, automation, analytics, and decision-support capabilities for operational relevance, technical maturity, cyber risk, reliability, maintainability, explainability, human oversight, and fleet suitability.
- Evaluate AI-enabled tools for use cases such as network monitoring, cyber anomaly detection, event correlation, predictive maintenance, help desk automation, configuration compliance, system health monitoring, log analysis, vulnerability prioritization, and operational decision support.
- Design test scenarios that reflect Navy afloat operating conditions, including limited bandwidth, disconnected operations, contested cyber environments, cross-domain constraints, variable data quality, and platform-specific operational limitations.
- Define data requirements, ground truth methods, evaluation datasets, labeling approaches, validation methods, and performance baselines for AI-enabled capabilities.
- Assess AI model performance using appropriate metrics such as accuracy, precision, recall, false positive rate, false negative rate, latency, robustness, drift, confidence calibration, explainability, and operational impact.
- Evaluate risks associated with hallucination, model brittleness, adversarial manipulation, data poisoning, prompt injection, bias, over-reliance, model drift, cybersecurity exposure, and failure modes in operational environments.
- Support AI red teaming, cyber survivability assessment, adversarial testing, safety reviews, and responsible AI evaluation activities.
- Develop human-machine teaming concepts, operator-in-the-loop workflows, trust calibration approaches, escalation procedures, and recommended guardrails for AI-enabled tools.
- Produce technical reports, evaluation findings, executive summaries, test observations, data analysis products, and recommendations for Navy and CACI leadership.
- Collaborate with systems engineers, cybersecurity engineers, software developers, data scientists, network engineers, operational testers, fleet users, and government stakeholders.
- Support technical interchange meetings, design reviews, test readiness reviews, operational assessments, demonstrations, and acquisition decision support.
- Provide SME input on AI governance, responsible AI implementation, model lifecycle management, configuration control, sustainment, monitoring, and continuous evaluation.
REQUIRED QUALIFICATIONS:
- Bachelor’s degree in computer science, data science, artificial intelligence, engineering, mathematics, statistics, cybersecurity, operations research, information systems, or a related technical discipline preferred. Advanced degree preferred.
- Additional years of directly relevant AI evaluation, test, cybersecurity, Navy, or DoD mission system experience may be considered in lieu of a degree.
- Demonstrated experience evaluating AI, machine learning, data analytics, automation, or decision-support systems in defense, intelligence, cybersecurity, network operations, enterprise IT, or mission system environments.
- Strong understanding of AI / ML evaluation methods, test design, performance metrics, validation approaches, model limitations, and operational risk assessment.
- Experience developing test plans, evaluation frameworks, measures of effectiveness, measures of performance, data collection plans, and technical reports.
- Familiarity with cybersecurity, enterprise networks, tactical networks, system monitoring, anomaly detection, log analytics, or network operations use cases.
- Ability to assess AI-enabled systems in operationally constrained environments, including limited bandwidth, degraded connectivity, edge computing, and mission-critical infrastructure.
- Understanding of responsible AI concepts, including transparency, explainability, human oversight, robustness, security, bias, accountability, and lifecycle monitoring.
- Experience working with cross-functional engineering, cyber, data science, software, test, and government stakeholder teams.
- Strong written and verbal communication skills, including the ability to brief complex AI evaluation findings to technical and non-technical audiences.
- Active DoD Secret clearance.
DESIRED QUALIFICATIONS:
- Experience supporting Navy, DoD, tactical edge, afloat, C4I, cyber, enterprise IT, or mission command systems.
- Familiarity with CANES, Navy afloat networks, ADNS, NAVWAR programs, RMF, cyber survivability testing, operational test, developmental test, or fleet experimentation.
- Experience evaluating generative AI, large language models, retrieval-augmented generation, autonomous agents, AI-assisted cyber tools, AI-enabled network operations, or predictive analytics systems.
- Knowledge of DoD responsible AI guidance, NIST AI Risk Management Framework concepts, RMF, Zero Trust, DevSecOps, MLOps, model monitoring, or secure software supply chain practices.
- Experience with data analysis tools, scripting, statistical evaluation, dashboards, test automation, or model performance analysis.
- Experience with AI red teaming, adversarial ML, cyber test events, operational assessments, or acquisition decision support.
- Top Secret clearance or SCI eligibility.
Requirements of position: Think analytically, effective verbal and written communication skills, make decisions, observe/remember details, interpret data, concentrate on tasks, adjust to change, handle stress/emotions. Regular attendance, maintain work schedule, attend meetings, meet deadlines, keyboard/type, handle confidential information, use math/calculations, stay organized, operate office equipment, may direct others. May be exposed to dust/dirt, humidity, and noise
Foxhole Technology is an Equal Opportunity Employer and makes hiring decisions without regard to race, color, religion, sex (including pregnancy, childbirth and sexual orientation), national origin, age, disability, genetic information, military/veteran status, or any other protected class.
Similar jobs
Est. 110,000 USD
Work Arrangement: Hybrid Clearance: Active Secret w/TS Capability Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized leader in navigating…
Est. 77,340 USD
About Us DLH delivers improved health and national security readiness solutions for federal programs through science research and development, systems engineering and integration, and digital transformation. Our experts…
Est. 95,000 USD
About Us DLH delivers improved health and national security readiness solutions for federal programs through science research and development, systems engineering and integration, and digital transformation. Our experts…
Est. 141,000 USD
Barbaricum is a rapidly growing government contractor providing leading-edge support to federal customers, with a particular focus on Defense and National Security mission sets. We leverage more than 17 years of support…
Est. 85,000 USD
About Us DLH delivers improved health and national security readiness solutions for federal programs through science research and development, systems engineering and integration, and digital transformation. Our experts…
Est. 144,000 USD
Electrosoft Services, LLC is an award-winning company that provides comprehensive technology-based solutions and services to federal customers. While cybersecurity is our specialty, we also focus on ICAM, Zero Trust, dig…
Est. 155,000 USD
About Us AGE Solutions is a premier technology and professional services company, providing in-depth consulting, advanced technology solutions, and essential services throughout the U.S. government, defense, and intellig…
Est. 140,000 USD
Company Description Air is the leader in Enterprise Readiness. Our mission is to establish readiness as a real-time condition that is continuously achieved. Today, a dangerous Readiness Gap exists between what the front…
Est. 147,500 USD
Work Arrangement: Hybrid (1-2 days onsite per week) Clearance: Active Secret Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized leader in…
Est. 144,000 USD
Company Description Air is the leader in Enterprise Readiness. Our mission is to establish readiness as a real-time condition that is continuously achieved. Today, a dangerous Readiness Gap exists between what the front…
Est. 124,000 USD
North Point Technology is looking to hire a Systems Engineering Architect to support AWS Resiliency and Recovery. The qualified candidate must currently hold an active Top Secret security clearance (with the ability to o…
Est. 185,000 USD
Work Arrangement: Hybrid Clearance: Active DoD Secret Start: Based on Contract Award Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized le…
Est. 118,000 USD
Company Description Air is the leader in Enterprise Readiness. Our mission is to establish readiness as a real-time condition that is continuously achieved. Today, a dangerous Readiness Gap exists between what the front…
Est. 120,000 USD
About Us AGE Solutions is a premier technology and professional services company, providing in-depth consulting, advanced technology solutions, and essential services throughout the U.S. government, defense, and intellig…
Est. 140,000 USD
Barbaricum is a rapidly growing government contractor providing leading-edge support to federal customers, with a particular focus on Defense and National Security mission sets. We leverage more than 17 years of support…
Est. 195,000 USD
MKS2 Technologies, LLC, an award-winning high growth small business, creates innovative and customer-centric technology solutions in the areas of Cyber Security, Instructional Design and Training, Software Engineering an…
Est. 200,000 USD
Work Arrangement: Hybrid Clearance: Active DoD Secret Start: Based on Contract Award Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized le…
Est. 124,000 USD
Company Overview:ANSER enhances national and homeland security by strengthening public institutions. We provide thought leadership for complex issues through independent analysis, and we deliver practical, useful solutio…
Est. 175,000 USD
Company Overview:ANSER enhances national and homeland security by strengthening public institutions. We provide thought leadership for complex issues through independent analysis, and we deliver practical, useful solutio…
Est. 95,000 USD
Empower, Innovate, Impact! At Team A-TEK, we EMPOWER people to drive INNOVATION that IMPACTS mission! A-TEK operates at the intersection of mission and innovation by applying our deep domain expertise across the federal…
Est. 205,000 USD
C3 AI (NYSE: AI), is the Enterprise AI application software company. C3 AI delivers a family of fully integrated products including the C3 Agentic AI Platform, an end-to-end platform for developing, deploying, and operat…
Est. 95,000 USD
About Us DLH delivers improved health and national security readiness solutions for federal programs through science research and development, systems engineering and integration, and digital transformation. Our experts…
Est. 135,000 USD
Work Arrangement: Hybrid (Currently 1-2 days per week onsite) Clearance: Secret Foxhole Technology provides robust cybersecurity and IT support capabilities for federal civilian and defense agencies. A recognized leader…
Est. 172,500 USD
MORSE Corp is an employee owned, small business based in Cambridge, MA, Arlington, VA, and Seattle, WA with a history of fielding cutting-edge technology. MORSE boasts a specially selected team of scientists, engineers,…
Est. 153,500 USD
As Sr. AI Engineer I, you’ll design secure, scalable, cloud-enabled architectures that modernize legacy Marine Corps systems and enable AI-assisted code conversion. This role partners closely with customers, engineers, a…
Est. 130,000 USD
About Us AGE Solutions is a premier technology and professional services company, providing in-depth consulting, advanced technology solutions, and essential services throughout the U.S. government, defense, and intellig…
Est. 170,000 USD
Overview Black Canyon Consulting (BCC) seeks a AI/Machine Learning Specialist that will lead design and deployment of AI/ML models that advance MC&FP’s mission outcomes while ensuring ethical and compliant use. We at…
Est. 161,050 USD
cBEYONData a subsidiary of SMX is seeking a Full Stack AI Developer to design, develop, and maintain scalable AI-enabled web applications and cloud-native solutions leveraging AWS technologies. The ideal candidate is a s…
Est. 120,000 USD
Barbaricum is a rapidly growing government contractor providing leading-edge support to federal customers, with a particular focus on Defense and National Security mission sets. We leverage more than 17 years of support…
Est. 140,000 USD
Barbaricum is a rapidly growing government contractor providing leading-edge support to federal customers, with a particular focus on Defense and National Security mission sets. We leverage more than 17 years of support…