Mandatory Procedure
MP5301.9001(b) -
Clearance: Multi-Functional Independent Review Teams
[ Revised April 6, 2015 ]
(INTERIM CHANGE: See Policy Memo 15-C-02)
1. Policy. The Clearance Approval Authority (CAA) must use Multi-functional Independent Review Teams (MIRT) as an integral component of the clearance process in 5301.9001(a) by validating each critical decision point (CDP) as described in paragraph 3.2 of this mandatory procedure.
2. Applicability. This procedure is required for all competitive acquisitions meeting the requirements of 5301.90 when contract values are $50M or more, including task orders for services issued against multiple award indefinite delivery/indefinite quantity contracts (to include GSA schedules) unless waived (see paragraph 5 below). At the discretion of the CAA, this mandatory procedure may be used for competitive acquisitions below $50M or for non-competitive acquisitions at any dollar threshold.
3. Independent Review. The CAA must establish an independent and objective process, employing cross-functional subject matter experts (SME) with source selection experience and knowledge of current source selection procedures, to constitute a MIRT. The MIRT operates in an advisory manner in order to foster frank and candid discussions regarding the soundness of the business and contracting approaches employed in the acquisition and the quality and consistency of source selection team products, independent of legal and clearance reviews. The results of these discussions (solutions, lessons-learned, etc.) should be shared with acquisition organizations across the Air Force.
3.1. Multi-functional Independent Review Team (MIRT). The MIRT is formed at the beginning of each competitive acquisition with membership approved by the CAA. When DAS(C) or ADAS(C) is the CAA, AQC will rely on the MIRT appointed by the MAJCOM/DRU/HQ AFICA/AFRCO SCO (or for AFLCMC and SMC, the SCCO). The CAA may use existing Independent Review Teams, peer reviews, Acquisition Center of Excellence (ACE) established review teams, or other established processes to satisfy this requirement. Note 1 To promote consistency, it is desirable that the same MIRT members participate in each critical decision point review for the duration of the acquisition unless otherwise approved by the CAA. The members of MIRTs must not be members of the source selection/competitive acquisition team. The team must be comprised of government personnel, to the maximum extent practicable.
The MIRT must have representation from:
-- Technical or Requirements Activity (e.g., engineering, program management, A1, A5, etc.)
-- Legal (e.g., other program counsel, SAF/GCQ, AFLOA/JAQ, etc.)
-- Contracting (.e.g., Contracting Officer, Chief of Policy, Technical Director/Advisor, etc.)
-- Others, as applicable (e.g., finance/cost, small business, contract management (DCMA) etc.)
Inclusion of peers from other bases, MAJCOMs/DRUs/FOAs, or other DoD Agencies provides a broader approach to the independent review process and facilitates the sharing of lessons-learned and best practices.
3.1.1. Independent Review Structure. The CAA must structure the independent review in a way that best achieves the objectives of the business and contract clearance process and tailored to the needs of the acquisition. The CAA must engage with the SSA upon establishment of the MIRT to promote an understanding of the MIRT process and explain the CAA’s rules of engagement for the MIRT (e.g., how many reviews are planned -- and if any specific CDP reviews are to be waived -- based on acquisition complexity, member experience and other oversight mechanisms, the planned duration of each review, and what is required to close MIRT comments prior to the CAA granting clearance), as well as the relationship between the clearance process and source selection process events. The MIRT process must focus on reviewing CDPs in a thorough and substantive manner where SMEs are granted the level of access to acquisition/source selection documents necessary to provide the CAA with a clear assessment of the soundness of the approach and methodology employed. The CAA should ensure that the established review process encourages discussion among MIRT members concerning their observations and recommendations.
3.1.2. MIRT Work Product. The MIRT must provide an assessment to the CAA on the state of the source selection/procurement. At a minimum, the MIRT must out brief the source selection/acquisition team at the conclusion of each CDP conducted. Documentation of MIRT reviews is required for the official contract file IAW paragraph 4. All MIRT and ACE comments must be adjudicated by the CAA. The SSA will be provided feedback by the CAA prior to the business or contract clearance approval.
3.2. Critical Decision Points (CDP). CDPs are precursor actions required to obtain approval for formal clearance events (e.g., approval to issue the solicitation (3.2.1.1 Pre-Business Clearance CDP)), or approval for the SSA to make the decision to award without discussions, approval for the SSA to request final proposal revisions, or for the SSA to make a source selection decision (3.2.1.2 Pre-Contract Clearance CDPs). These CDPs are conducted by the MIRT prior to obtaining the appropriate business or contract clearance from the CAA.
3.2.1 CDPs.
3.2.1.1 Pre-Business Clearance CDPs.
-- Review draft ASP Brief Note 2 or review of draft AP Note 3 (Includes review of requirements documents, results of risk assessment, incentive structure, as applicable.)
-- Review Sections L and M of the Request for Proposal (RFP) Note 4 (Includes review of Source Selection Plan and requirements documents to ensure proper evaluation criteria.)
3.2.1.2. Pre-Contract Clearance CDPs.
-- Review draft Competitive Range Brief or review of draft Award without Discussions Brief (Review of these draft briefs may include review of supporting documentation and evaluation notices or interim ratings, etc.)
-- Review draft Final Proposal Revision (FPR) Request (Review of this draft brief may include review of pre-FPR brief including interim ratings after discussions, etc.)
-- Review draft Source Selection Decision briefing (Review of this draft briefing may include review of pre-source selection decision brief, PAR, etc.)
3.2.1.3. Areas of Special Interest (ASI). The CAA may require the MIRT to assess ASI in addition to the CDPs discussed in the previous paragraphs (e.g., review of source selection plans or debriefing charts/script, the conduct of mock debriefs, review of model contracts and all attachments). The MIRT also has the discretion to review ASIs that may impact their assessment of the CDPs.
4. Documentation. Documentation that independent reviews were conducted IAW this procedure must be included in the contract file under the business and contract clearance tabs. Also include the actions taken to adjudicate the MIRT comments by the CAA. The documentation is considered “Source Selection Information” and/or “For Official Use only” and must be handled IAW FAR 2.101 and 3.104.
5. Waiver. The CAA may waive the use of MIRTs in support of a specific clearance action for individual acquisitions or specific CDPs based on acquisition/source selection history and procurement/source selection experience of the acquisition team (i.e., recurring nature of the requirement, no history of sustained protest in a competitive acquisition, and successful experience with evaluation criteria in a competitive acquisition).
Additional Considerations When Forming A MIRT
3.1.1 Other MIRT Considerations
1. Operational Considerations
a) Identification of content of required documentation such as briefing charts, record of reviews, adjudication/disposition of review comments;
b) Notification procedures for MIRT reviews;
c) Method of feedback to source selection team, to the CAA, and if applicable to the SSA;
d) Procedures for acquisition team and the MIRT to share and exchange information during the review (i.e., discussions that may clarify and remedy a concern on the spot; opportunity to pass on thoughts or recommendations for consideration when drafting RFPs for future source selections);
e) Appropriate forum to conduct the MIRT reviews (i.e. roundtable, telecom, video conferencing); or,
f) Procedures for the source selection team to provide the CAA with its response to the findings of the MIRT to include procedures for adjudication of a MIRT finding that the RFP is fatally flawed (i.e., desired intent of RFP, the need to clearly demonstrate that the proposed RFP correction will not favor a particular offeror).
2. Teaming/Membership Considerations
a) Whether to establish standing or ad hoc MIRTs or to assign members based on unique circumstances of each acquisition. Beware of the possibility that continuous membership may lead to members losing their independence or objectivity over time and getting pulled into the “group think” scenario;
b) Whether the team should be led by one individual or co-chaired
3.2.1.3 The review process may include the following areas:
1. Pre-Business Clearance CDP
a) Pre-ASP
Source Selection Team Membership
-- Are there appropriate, knowledgeable, and qualified source selection team members, to include a sufficient number, assigned?
-- Did the briefers effectively articulate their decisions/justifications?
ASP Brief/Draft Acquisition Plan
-- Is the requirement captured by performance-based objectives?
-- Is the requirement defined to capture the minimum program objectives and can it be realistically proposed to?
-- Are the requirements documents (including operational capability requirements) translatable into appropriate, evaluative criteria?
-- Does the market research support the requirements, acquisition strategy and recommendations?
-- Does the risk analysis include contractors’ risk as well as government’s?
-- Does the risk analysis address and support the recommendations of contract type, pricing structure, type of source selection?
-- Were other contractual vehicles considered, e.g., GSA, multiple award contracts, other DoD contracts, etc.? Was a cost/benefit analysis performed?
-- Are there special contract requirements?
-- Are potential Organizational Conflict of Interest (OCI) issues addressed?
-- Are the milestones established by focusing on the key events and with realistic dates to accommodate accomplishing the actions required at the key events? (e.g., is the source selection schedule event driven and not schedule driven?)
-- Do the milestones include review/participation by a MIRT and OSD Peer Review, if required?
-- Does the briefing support the recommendations, e.g., type of contract, pricing structure, type of source selection, etc.
-- Is the acquisition strategy executable?
-- If using ASP as an acquisition plan – are there sufficient details, either in the brief or in the notes – that the ASP can be approved as the acquisition plan?
-- If there has been any consolidation of contracts, was the bundling and consolidation analysis performed prior to deciding on an acquisition strategy? Is the basis for bundling or consolidation justified?
-- If this is a single-award IDIQ, do any of the exceptions at 10 USC 2304a(d) apply?
b) RFP Sections L and M
-- Are the individual elements to be evaluated under each factor/subfactor stated in a clear, non-complex manner?
-- Are evaluation criteria appropriate (e.g., limited key discriminators based on risk analysis) in order to select the best value contractor?
-- Are cost considerations appropriate for strategy? If one of the criteria is to evaluate Most Probable Life Cycle Costs (MPLCC) this is a “red flag” – were lessons-learned from previous source selections where MPLCC was used discussed before recommending it as a criteria? Was required approval obtained? Was rationale and methodology thoroughly explained and executable?
-- If most probable cost (MPC) is being evaluated – is it only for the instant contract?
-- Was a side by side comparison of Sections L & M made to ensure how what is being asked for will be used in the evaluation process?
-- If using a tradeoff source selection approach, does the evaluation identify and prioritize specific trade space or performance thresholds for which the government is willing to pay more? Is it clear from the evaluation criteria what the offeror would provide that the government would constitute more advantageous?
-- Are minimum thresholds and maximum performance objectives clearly defined?
2. Pre-Contract Clearance CDP
a) Pre-Initial Evaluation/Pre-Competitive Range (Award w/o Discussions)/Pre-release of Interim Ratings
Evaluation
-- Did the evaluation team follow/comply with:
--- Sections L&M of the RFP?
--- Source Selection Plan?
-- Was there consistency and fairness in evaluating each offer against Section M (evaluation criteria)
-- Was there consistency in applying the ratings across the offerors?
-- What clarifications and/or communications, if any, were conducted?
--- Were they appropriate clarifications and/or communications?
--- Were meaningful and open communications conducted in which the offeror and government clearly understood each other’s position and assumptions? (It is not sufficient to handle communications in a way that is it only for the benefit of the government to understand the offeror’s proposal.)
-- Did the evaluation team apply any standards or undisclosed evaluation criteria to their evaluation of proposals or past performance?
Documentation
-- At a minimum, were deficiencies, weaknesses, significant weaknesses, and adverse past performance described in the evaluation notices? Were assumptions addressed as appropriate or acceptable?
-- Is supporting documentation describing details of the evaluations readily available during the briefing?
-- Does the supporting documentation (i.e., evaluation worksheets, initial evaluation report, draft PAR) fully, completely, and contemporaneously document the entire evaluation? Does the briefing accurately compare to the official documentation?
Briefing Charts
-- Does the briefing tell the story of proper evaluations from receipt of proposals, through the technical, past performance and cost evaluations, through the type of clarifications and communications conducted, through formations of evaluation notices, to the initial ratings/standings?
-- Does the content and quality of the briefing reflect Contracting officer and SSEB Chairperson(s) or equivalent review?
-- Does the briefing support the recommended competitive range (or to award without discussions)?
-- Did the briefers effectively articulate their decisions/justifications?
b) Pre-Final Proposal Revision Request Brief
Discussions
-- Were discussions conducted IAW FAR 15.306(d)?
-- Did the discussion process utilized achieve meaningful and transparent communications?
Documentation
-- Does the supporting documentation adequately describe the basis and justification for the ratings?
-- Do the Interim Ratings prior to final evaluation support the degree of discussions held with each offeror? (e.g. there is no misunderstanding by the offeror that the Contracting Officer clearly discussed the issues that resulted in the identification of significant weakness and/or any deficiency.)
-- Does the supporting documentation, fully, completely, and contemporaneously document the entire evaluation? Does the briefing accurately compare to the official documentation?
Briefing Charts
-- Does the content of the briefing bring the SSA up to date from approval to establish competitive range to completion of meaningful discussions (with or without releasing of Interim Ratings to the offeror)?
-- Does the briefing content confirm that meaningful discussions were conducted?
-- If interim ratings were not released prior to Final Proposal Revision request, has the Contracting Officer clearly articulated his/her rationale in the briefing charts?
-- If any offeror’s proposal should no longer be in the competitive range, does the briefing support the recommendation to eliminate it from consideration for award?
-- Did the briefers effectively articulate their decisions/justifications?
c) Pre-Source Selection Decision Brief
Documentation
-- Did the SSEB complete proposal evaluation, incorporate information provided through discussions and Final Proposal Revisions IAW section M?
-- Is the Proposal Analysis Report (PAR) reflective of the evaluations and justification documents?
-- If a SSAC was used, are the Comparative Analysis Report and recommendations complete?
-- Does the supporting documentation fully, completely, and contemporaneously document the entire evaluation? Does the briefing accurately compare to the official documentation?
Briefing Charts
-- Do the briefing charts clearly summarize and justify the evaluation results?
-- If Proposal Analysis Report is not used, does the briefing contain adequate details of the evaluation results and a comparative analysis (cost/price, past performance, technical risk, and a source selection recommendation) of the competitive offers?
-- Can the SSA make an integrated assessment best value decision based on the information presented in the briefing?
-- Can the information in the briefing be used, as applicable, for the source selection decision document?
-- Did the briefers effectively articulate their decisions/justifications?
3. Areas of Special Interest
a) Pre-Debriefs
-- If Sections L&M were complied with; if meaningful discussions were conducted with the offeror; and if offeror aware of its standings (via initial & interim ratings) throughout the evaluation process, the majority of the debriefing will focus on the final evaluation.
-- The content of the debriefing slides should be taken from those used for the offeror during the competitive range, interim ratings, and final proposal revision requests briefings. (The offeror should not be surprised with the content of this information during a debriefing.)
-- Purpose is to put the “debriefing” team in a mock situation to be able to handle offeror’s questions and comments.
-- Can the members of the debriefing team effectively articulate their findings and their justifications?
-- Does the debriefing focus on the government’s evaluation of the offeror’s proposals instead of the selected winner?
-- Will the debriefing assist/instruct the offeror on what it did well and what it could have done better?
********************************************
Note 1: Personnel participating as members of or advisors to source selection teams or procurement teams may not be an acceptable substitute for the independent review due to their lack of objectivity and independence. However, the CAA has the discretion to use “advisors” as members of a MIRT.
Note 2: In the event the contents of ASP do not align with FAR Part 7 and its supplements or an ASP is not held, review of written AP or equivalent documentation will be conducted.
Note 3: Applicable to Life Cycle Management Plan (LCMP), Acquisition Plan, and other similar documentation.
Note 4: References to RFP Sections L and M include equivalent solicitation provisions for the acquisition of commercial items.