AI Risk Assessment
Made AI risk legible to executives
Overview
- Role
- UX Engineer
- Period
- 2022 Q4 - 2023 Q2
What I did
- Conducted UX Research & Analysis
- Designed Core Workflows
- Researched & Designed Elicitation Process
- Developed UI & Report Visualizations and Interactions
- Built High-Fidelity Interactive Prototypes
- Conducted Usability Testing & Iteration
Problem
Enterprise companies struggled to systematically assess and monitor AI risks across their applications. Existing manual and inconsistent methods led to potential blind spots and difficulties ensuring regulatory compliance and ethical AI usage.
What I designed & built
We developed an automated platform leveraging machine learning to analyze AI system risks, enabling executives to make informed strategic decisions about their AI systems. The platform features an intuitive dashboard for risk visualization and clear reporting tailored for non-technical stakeholders.
Process
Research & Discovery
Conducted interviews across enterprise stakeholders (technical and non-technical) and AI research teams to define critical requirements and pain points with existing processes. Competitive analysis revealed key gaps in automation and reporting clarity within current industry tools, highlighting opportunities for our platform.
Existing AI risk assessment tools were either too technical for executives or too simplistic for comprehensive analysis.
- Non-technical stakeholders needed visual, intuitive risk summaries
- Technical teams wanted detailed, readable risk breakdowns
- Regulatory compliance required standardized risk assessment
Design Process
Mapped out the end-to-end user flow, designing an intuitive workflow for users to upload models and associated datasets. A key challenge was the elicitation process: I worked closely with the research team and designed a guided questionnaire to efficiently gather necessary contextual information about the AI system from users, ensuring the backend ML model received sufficient data for accurate risk assessment.
How to gather complex technical information from users and educatewithout overwhelming them or missing critical details?
- Users often don't know what information is relevant for risk assessment
- Long questionnaires lead to abandonment and incomplete data
- Context matters more than technical specifications for risk analysis
- Progressive disclosure keeps users engaged while gathering depth
Designed a progressive questionnaire that adapts based on previous answers. This approach helps to gather the necessary information from the users and inform the risk dimentions relevant to their own use case.
The final questionnaires have conditional tree structures, which was refined after multiple iterations with the researchers and the users.
UI Development
Designed a clean, data-driven platform including the dashboard visualizing overall risk scores and key risk dimensions breakdowns. Focused on creating detailed, yet easy-to-understand report layouts with clear data visualizations, specifically tailored for executive-level comprehension.
How to present complex, multi-dimensional risk data in a way that's immediately actionable for both technical and executive audiences?
- Executives need high-level risk scores with ability to drill down
- Color coding is more intuitive and consistent across all views
- Risk context is as important as the score itself
- Comparison capabilities help users understand relative risk levels
- Traditional tabular data presentation
- Single overall risk score with minimal breakdown
- Separate dashboards for different risk categories
Created a layered information architecture with visual hierarchy, overall risk score prominently displayed, with expandable sections for detailed breakdowns. Used familiar traffic light colors with additional context.
Tested multiple different visualization approaches, final design combined the clarity of simple charts with the depth of detailed breakdowns.
Key decisions
Color coded risk score with contextual explanations
Users immediately understood risk levels but also needed context for what each level meant
Reduced effort and time to understand risk assessment
Contextual help vs. simplified language
Users needed to maintain credibility with technical stakeholders while understanding the content themselves
Balanced professional terminology with explainability, which increased user confidence
User Testing
Conducted moderated usability testing sessions with target users (e.g., risk managers, AI developers) using high-fidelity Figma prototypes. Feedback led to several key iterations, particularly clarifying risk terminology, resulting in higher user satisfaction.
Initial prototypes tested well for functionality but users struggled with risk terminology and report interpretation.
- Users confused 'bias risk' with 'model bias', needed clearer definitions
- Risk scores without context felt arbitrary and unhelpful
- Users wanted to compare risks across different models
- Export functionality was critical for stakeholder communication
- Provide glossary as separate reference document
- Use technical terminology consistently throughout
- Simplify language at the cost of precision
Implemented contextual tooltips and inline explanations that appear on hover/click. This maintains professional terminology while providing immediate clarification.
Shipped outcome
This project validated key concepts for automating AI risk assessment through an interactive prototype. By making complex processes tangible, it provided a clear vision for future tools and effectively engaged stakeholders in strategic discussions about responsible AI implementation. Notably, the prototype resonated strongly with a key client, whose positive feedback was instrumental in the decision to pursue further development based on these concepts. The insights gained informed both this next step and broader potential development pathways for AI governance solutions.
Learnings
- The importance of monitoring in AI lifecycle
- How to make the complex and technical process of AI risk assessment accessible to non-technical stakeholders
- How to effectively visualize complex risk metrics
Next steps
- Implement automated mitigation suggestions
- Expand coverage to more AI frameworks
- Develop custom risk assessment templates