ELATE™

Evidence-based List of Exploratory Questions for Artificial Intelligence Trust Engineering

Open Call

We are currently looking for development teams to help us pilot new ELATE™ tools! Please reach out if interested: elate@mitre.org.

Contacts

Email: elate@mitre.org

Links

ELATE Latest Version
A list of 50+ exploratory questions for promoting critical thinking, with real-world examples.

Link

ELATE Cards
Digital and printed card decks for applying ELATE in team discussions and agile ceremonies.

Link

Technical Report
ELATE technical report with full description of approach and methodology.

Link

What is ELATE?

ELATE empowers teams to ensure trustworthy artificial intelligence (AI) products throughout their development cycle. The toolkit consists of 50+ exploratory questions for teams to prioritize and answer regularly, illustrated by actual incidents involving AI.

Currently, developing “Trustworthy Artificial Intelligence (AI)” hinges upon top-down, principles-based approaches, which are difficult to put into practice. We developed ELATE to help AI developers take a bottom-up approach to developing trustworthy AI.

We collected data from multiple sources about incidents where people gained or lost trust in AI “in the wild” and developed a list of items to consider, each including exploratory questions and evidence from real-world examples.

Why Use ELATE?

It is difficult to predict how AI will interact with a sociotechnical system. ELATE codifies hindsight to increase foresight.​

ELATE generates actionable outcomes that drive common systems engineering processes (requirements, designs, etc.).

ELATE Cards

Use ELATE cards during agile ceremonies to prompt critical thinking among developers, users, and stakeholders, to generate requirements, designs, test cases, etc. ELATE cards should be used with 4-10 developers, users, and stakeholders.

A stylized image of ELATE Cards. Three cards are fanned out on top of abstract iconography of tables and people. The first card shows all 8 ELATE focus areas with the names, short descriptions, and colors. The second card reads: "Stakeholders in Development. How might users, domain experts, and other stakeholders best be involved in the development of the AI (early and often enough)? How might non-users impacted by the AI have a voice in its development? Example: Users lost trust in an AI where end users were not involved in the conceptualization and design of the AI: “They missed a huge part… and they did not include the [users] enough… Sometimes it needs to include less technical focus and more involvement of the people who make these decisions every day without computers.” The third card shows the back of the ELATE card and is mostly hidden by the other two.
Image Gallery (Click for full size)
Closeup photograph of the MITRE ELATE Cards unboxed. The box is on the right with title in the center: MITRE ELATE. The text at top of the box: "An evidence-based resource that supports the engineering of more trustworthy AI". And at the bottom: "A collection of 50+ real-world examples where people gained or lost trust in AI, with exploratory questions to drive critical thinking about your own AI". On the left is the deck of cards fanned out. Some cards are gray, others orange, green, purple, and red. The card on top is red and has the text: "High-Risk / Low-Frequency. How might a single erroneous output from this AI cause a big problem for someone? How might the AI enable preventing this before it happens, and/or make it easier to reverse the impacts? Example: Users acted on a contraceptive app's predictions, leading to unwanted pregnancy". A citation number follows the text. The red at the top of the card has an icon and the name of the ELATE focus area containing this card: "Controls & Safeguards. Preventing harms to users and others."
Photograph of the ELATE Card deck and Info Booklet lying on top of the disassembled box, showing text on the inside of the box. The text closest to the camera and most readable says: "CONOPS: Implications for how the AI is deployed, maintained, administered; how users need to be trained, etc. These can be captured in a variety of artifacts."
Photograph of the MITRE ELATE Cards unboxed. The box is on the right with title in the center: MITRE ELATE. The text at top of the box: "An evidence-based resource that supports the engineering of more trustworthy AI". And at the bottom: "A collection of 50+ real-world examples where people gained or lost trust in AI, with exploratory questions to drive critical thinking about your own AI". On the left is the deck of cards fanned out. Some cards are gray, others orange, green, purple, and red. The card on top is red and has the text: "High-Risk / Low-Frequency. How might a single erroneous output from this AI cause a big problem for someone? How might the AI enable preventing this before it happens, and/or make it easier to reverse the impacts? Example: Users acted on a contraceptive app's predictions, leading to unwanted pregnancy". A citation number follows the text. The red at the top of the card has an icon and the name of the ELATE focus area containing this card: "Controls & Safeguards. Preventing harms to users and others." At bottom, additional cards are fanned out. toward the camera. These cards are blue with the focus area header "User Requirements: Ensuring the AI meets user needs." The text on the top blue card reads: "Stakeholders in Development: How might users, domain experts, and other stakeholders best be involved in the development of the AI (early and often enough)? How might non-users impacted by the AI have a voice in its development? Example: Users lost trust in an AI where end users were not involved in the conceptualization and design of the AI: “They missed a huge part… and they did not include the [users] enough… Sometimes it needs to include less technical focus and more involvement of the people who make these decisions every day without computers.” In the middle of the photo toward the camera is a card listing all the ELATE focus areas in their colors. To the left of the photo toward the camera is the ELATE Cards Info Booklet.

TERMS OF SERVICE

PUBLIC RELEASE

Approved for Public Release; Distribution Unlimited. Public Release Case Numbers: 23-00018-2; 24-00712-2.

LICENSE

The MITRE Corporation (MITRE) hereby grants you a non-exclusive, royalty-free license to use MITRE ELATE™ for research, development, and commercial purposes. Any copy you make for such purposes is authorized provided that you reproduce MITRE’s copyright designation (shown below) and this license in any such copy:

“© 2024 The MITRE Corporation. This work is reproduced and distributed with the permission of The MITRE Corporation.”

DISCLAIMERS

ALL DOCUMENTS AND THE INFORMATION CONTAINED THEREIN ARE PROVIDED ON AN “AS IS” BASIS AND THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE MITRE CORPORATION, ITS BOARD OF TRUSTEES, OFFICERS, AGENTS, AND EMPLOYEES, DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION THEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.