Prior to starting this process, a potential Challenge Sponsor will have reached out to the AFWERX Challenge team to express an interest in sponsoring a Challenge. That will then kick off initial discussions with the government team for AFWERX Challenge.
The discussion usually covers the problem definition, the solution ideas, other efforts that have been made to solve the problem, an assessment of the suitability of non-traditional industry to provide solutions, a high-level timeline, and finally, funding and contracting considerations for prototyping and transition. If those conversations go well, then the next phase is the scoping discussion.
Here follows the overview of the Challenge Process from the government perspective. Please note that even though every Challenge is different and some aspects of a Challenge may be unique, they all follow the same high-level process and each phase is always completed in one form or another in a sequential order.
This initial phase of the Challenge Process is with the Challenge Sponsor team, the government members of AFWERX Challenge and the contractors that may be supporting the Challenge. This is usually a 2-3 hour meeting to cover the objectives of the Challenge, the need identified, the impact of solving the problem, the timeline for the Challenge phases, contracting pathways, etc.
The outcome of the Scoping Discussion is:
A Challenge Approval Form that is the formal request to AFWERX for the creation of a Challenge. If the Challenge is approved, the contracts are put in place and so is the funding for the Challenge facilitation.
The detailed project plan and timeline to execute the Challenge, including the date of the Challenge Definition Workshop.
Once the contractor team receives Authority To Proceed (ATP) then the Challenge Discovery phase starts. This phase covers:
- Research to better understand the problem, the outcome, and potential solutions
- Research to identify Subject Matter Experts in the areas covered by the Challenge
- Research to identify any industry associations that may be able to help spread the word of the Challenge to people interested in either attending the Challenge Definition Workshop or submitting a solution to the Crowdsourcing phase.
- Outreach to people that may be interested in attending the Challenge Definition Workshop
Challenge Definition Workshop
The first time potential non-government stakeholders find out about the Challenge topic is when the Challenge Definition Workshop is advertised. This advertising is usually by way of email communication and social media posts letting people know about the topic and asking people to register their interest in attending the workshop.
The workshop is usually promoted for at least 4 weeks (preferably 6-8 weeks) prior to the workshop date to give people time to register their interest, be invited, and then make travel arrangements to attend the workshop (if it is an in-person event).
Challenge Definition Workshops are design thinking workshops where subject matter experts from industry, academia, and the Air Force are guided by the facilitation team and Challenge Sponsors to explore the problem, define the outcome, identify solution components and finally, craft the Challenge Statement for the Challenge Crowdsourcing Campaign.
These workshops were usually two-day in-person sessions, but due to COVID travel restrictions, some are now virtual or hybrid. If virtual, they are typically 3-day workshops.
During the workshop, participants collaborate and iterate to:
- Create a Challenge Problem Statement - What is the problem the Air Force is trying to solve?
- Create a Challenge Outcome Statement - What is the outcome the Air Force would like to get to?
- Identify Solution Components for the Challenge - What are the potential parts of the solution?
- Identify the Focus Areas of the Challenge - What are the types of solutions that could solve the problem?
- Define Outreach Goals - Which types of entities that could solve the problem or industry associations that could promote the Challenge should be invited?
Typically at the end of the workshop, the Challenge Sponsor team has a different perspective of the problem, outcome, and solution ideas related to the topic. The AFWERX Challenge and the Challenge Sponsor Team then meet to agree on the following:
- Finalization of the Evaluation Criteria and the Rubric to assess submissions so that they can be shared in the Challenge Statement
- Finalization of the Challenge Statement that is about to be launched
- Plan for the recruitment of the evaluation team of subject matter experts
Once the objectives for the Challenge Discovery Workshop are met, the Challenge Statement is finalized and the Crowdsourcing Campaign is ready to launch. The campaign is usually 6-8 weeks, meaning that this is the timeline within which industry can submit solutions and the evaluators are recruited.
During this Crowdsource Campaign phase, the following activities take place:
Marketing of the Challenge and specific outreach:
AFWERX Challenge performs extensive market research, runs marketing campaigns, and partners with industry associations to do networked outreach. The goal is to get as many quality submissions from industry and academia as possible.
Webinars & Networking Events:
During the crowdsource campaign period, a series of webinars and networking events take place. The purpose of these events is to:
Better inform industry and academia about the goals of the Challenge through presentations and Ask Me Anything sessions that cover Challenge objectives, submission guidelines, evaluation criteria, and more.
Offer networking opportunities to potential submitters to the Challenge to offer the potential for collaboration opportunities to create even more powerful submissions.
During the Crowdsource Campaign, solution providers are invited to join the AFWERX Challenge site to learn more about the Challenge and get to assess their suitability to submit a solution.
Solutions can be submitted in draft form first and then finalized any time prior to the end of the submission period.
Evaluator recruitment and training
During the Crowdsource Campaign, the AFWERX team and Challenge Sponsor work to identify and recruit a team of subject matter experts who can evaluate the submissions at the conclusion of the crowdsourced campaign.
Most Challenges will have an evaluation team of 50-100 subject matter experts.
Training for the evaluation team on the evaluation platform and the criteria themselves will usually take place the final week of the Crowdsource Campaign.
As soon as the Crowdsource Campaign submission window closes, the evaluations can begin. The goal is to complete all the submissions within a two-week time period.
AFWERX Challenge requires a minimum of three individual evaluations of each submission, although our preference is for five evaluations. This means that if there are 100 submissions to a Challenge, there will be a minimum of 300 evaluations or three per submission. Evaluations are mainly quantitative, however, there is also an opportunity to provide qualitative feedback.
The AFWERX team provides regular reports to the Challenge Sponsor team throughout this two-week period to ensure that the goals of the evaluations are met. Once the evaluations have been completed, the results are assessed using business intelligence and analytics, a series of statistical reports are created to provide the Challenge Sponsors with the maximum amount of information available to make their selections for the Showcase. Typically, the top 10-20% of submissions are invited to the Showcase.
Once the Challenge Sponsor team finalizes the invitations they would like to make for the Showcase, the AFWERX team immediately notifies both the selected teams and those not invited.
The Showcase Phase usually starts with a webinar to explain the next steps to all invited teams. Given the current everchanging COVID situation, the showcase could be virtual, hybrid or in-person. Regardless of form, the Showcase will have three phases:
Networking events are organized to promote collaboration between teams for either the benefits of the Challenge or other potential opportunities. In previous Challenges, there have been teams that partnered together to receive prototype contracts and provide solutions to the Air Force.
Each solution submission selected will pitch to the evaluation team and answer questions. The pitch will usually be 5-10 minutes and then a 10-15 minute Q&A. However, these times vary depending on the amount of teams selected and the number of evaluators that are available.
Finally, there is a trade show event where each team has a booth which is visited by DoD members, other solution providers, and other attendees. This is obviously very different if the event is virtual or in-person.
Contracting & Prototyping
At the conclusion of the Showcase phase, the next phase is contracting and usually prototype contracting. Here, a Request for Proposal is sent from either the Challenge Sponsor or any other DAF member if a Commercial Solutions Opening (CSO) is in place.
Teams then respond with a Statement of Work and what follows are a series of negotiations prior to contracts being executed and funding transferred. Then, the prototyping phase starts with teams likely being given time, funds, and access to Air Force resources to develop prototype solutions.
The ultimate goal of any Challenge is to have solutions transitioned to the Air Force to improve the efficiency, effectiveness and quality of life of our Airmen and Guardians.