← Back

TL;DR

Research Objective

Research Methods Approach

Findings

Impact

Reflection

USABILITY CASE STUDY

USABILITY CASE STUDY

LAUNCHED 2026

LAUNCHED 2026

How College Students Use Foundry?

MY ROLE

UX Researcher

TEAM

4 Researchers

SPONSOR CLIENT

Foundry - Knowledge Team

TIMELINE

January - March 2026

SKILLS

User Research

Usability Testing

TL;DR

Overview

Microsoft Foundry IQ (formerly Azure AI Studio) is a service offering for enterprise AI operations, model builders, and application development. While powerful, Foundry's complexity creates a steep learning curve, especially for emerging users outside the enterprise world.

WHAT I SET OUT TO LEARN?

I led a usability study to evaluate whether college students with computer science backgrounds who are interested in using AI and startups could successfully create AI agents using Foundry's Knowledge and Data features.


Identify where the experience breaks down and give the Foundry team a severity-ranked roadmap to fix it.

RESEARCH OBJECTIVE

Research Questions

This study was designed to answer two core questions:

  1. Can a first-time user successfully create an AI agent and attach a knowledge source without external guidance?

  2. Where in the agent-creation workflow do users hesitate or misinterpret, and why?

Recruitment Process

I recruited 8 college students through the university network and local communities. The screening criteria required:

  • Basic familiarity and looking to leverage AI for their project

  • Has a computer science, technical, and/ or startup background

  • Comfortable working with datasets and navigating through developer tools and technical interfaces


The participants had different technical backgrounds and experience with competing tools, ranging from moderate to high levels.

GOAL

I wanted to test whether Foundry's usability barriers were universal or correlated with technical depth.

By conducting and understanding the platform, I was able to create an alignment test of key features

By conducting and understanding the platform, I was able to create an alignment test of key features

RESEARCH METHODS APPROACH

Heuristic Evaluation

Before making a study plan, I conducted a walkthrough of the full agent-creation flow, familiarize myself with Foundry product, and evaluating it against Nielsen's 10 usability heuristics. This audit surfaced potential usability issues, which I mapped across the workflow to form testable hypotheses.

Help and Documentations

No onboarding or contextual guidance for first-time users

Consistency and Standards

Labeling and technical terms are confusing and not intutivve

Labeling and technical terms are confusing and not intuitive

Navigation Issues

The playground page is complicated to navigate

Moderated Usability Testing

I conducted one-on-one 45-minute sessions where participants completed three tasks in the live Foundry environment with provided sample data:


  • Task 1: Create a new AI agent from the Foundry homepage

  • Task 2: Upload the provided data set and attach it to the agent

  • Task 3: Test the agent by asking it a question that requires the uploaded knowledge


I used a think-aloud protocol, asking participants to narrate their reasoning as they navigated. Sessions were screen-recorded with audio. We tested in the live environment because we needed to evaluate real system feedback, error states, loading behavior, and users' interactions.

I reframed the usability study as a strategic test focusing not just on task completion, but on how well the user experience on the product

NORMAN'S SEVERITY SCALE

4

Usability catastrophe: Prevents task completion; imperative to fix

3

Major problem: Causes significant confusion; important to fix

2

Major problem: Causes significant confusion; important to fix

1

Cosmetic: Surface-level issue; fix if time permits

I prioritized findings using Nielsen's severity scale. A confusing label is a different problem from a blocker preventing every user from completing the core task. Severity framing gave the Foundry team an actionable roadmap, not just a list of complaints.

FINDINGS

Five Issues. One that stop EVERYTHING

Usability Catastrophe

Guardrail Roadblock

8 out of 8 participants experienced a critical task failure, unable to proceed with agent creation due to an unassigned guardrail blocking the interface.

Short reel shows users got stuck with guardrail

Through our testing, we identified three usability failures:

  • Ambiguous System Status: The system left new agents in an unclear "inheriting" state rather than defaulting to a standard guardrail.

  • Misleading Error Pathways: When blocked, the system prompted users to "create a guardrail," when the actual resolution was to reassign an existing default.

  • Lack of Contextual Guidance: Participants could not view their active guardrail status or understand version differences, leading to frustration and wasted time.

RECOMMENDATIONS

  • Automatically assign default guardrail during agent creation so new users are never blocked at this step


  • Rewrite the error message: replace "Create Guardrail" with "Reassign Guardrail" and include a one-line explanation of why the interaction is blocked


  • Add visible guardrail status to the agent dashboard (e.g., a status badge showing "Default Guardrail v2.1 · Active")


  • Add tooltips explaining guardrail purposes and version differences

Major Problem

Unclear Terminology

7 out of 8 participants cited confusing terminology as the most persistent usability friction throughout their experience.

Short reel of users confusing "Knowledge" and "Tools"

  • The issue: Overlapping names caused immediate hesitation. "Tools" vs. "Knowledge" confused every technical participant, and "Agent Instructions" vs. "Message Agent" created a second layer of ambiguity.

  • The impact: While this did not result in outright task failure, the mismatched mental model (e.g., placing file uploads under "Tools") significantly slowed workflows, increased cognitive load, and undermined user confidence.

RECOMMENDATIONS

  • Conduct a terminology audit across the platform to ensure every label is distinct and descriptive


  • Add contextual definitions, either inline descriptions below section headers or tooltips on hover, for "Tools," "Knowledge," and "Agent Instructions."


  • Benchmark terminology against competing platforms (OpenAI, Google Vertex, Hugging Face) to identify where Foundry's labels deviate from emerging industry conventions

Minor Problem

Cluttered Landing Page & Visual Hierarchy

4 out of 8 participants experienced immediate discoverability issues upon landing on the Foundry homepage, delaying their time-to-task.

  • The issue: The intended primary CTA ("Start Building") was too small, while a secondary element ("Coding Quick Start") dominated the visual hierarchy.

  • The impact: Combined with terminology confusion, this visual imbalance caused a 50% hesitation rate, as users could not intuitively discern the correct starting point.

"I see like four things I could click. I'm not sure which one actually starts the process." — interviewee

RECOMMENDATIONS

  • Increase the size, contrast, and positioning of "Start Building". It should be the single dominant element above the fold


  • Demote "Coding Quick Start" to a secondary placement (e.g., below the fold or in a sidebar)


  • Reduce overall visual density on the landing page, fewer competing cards, clearer hierarchy

Minor Problem

Unguided Navigation

8 out of 8 participants deviated from the expected user flow when navigating the Foundry platform.

Short reel of users confusing "Knowledge" and "Tools"

  • The issue: After creating an agent, users were left without clear next steps. No guided onboarding, no progress indicators, no suggested workflow.

  • The impact: Users resorted to exploratory clicking; 2 of 8 participants clicked through every sidebar tab just to deduce the platform's structure. While everyone eventually found their way, the unguided approach increased cognitive load and time-to-task by an estimated 40–60%.

RECOMMENDATIONS

  • Add a post-creation checklist, success feedback or progress indicator: "Agent created ✓ → Add knowledge source → Test your agent → Deploy."


  • Introduce an optional guided walkthrough for first-time users (4 of 8 participants specifically requested this)


  • Surface contextual "next step" prompts after key actions (e.g., after agent creation, prompt: "Your agent is ready. Add a data source to make it smarter")

Minor Problem

Misleading Error Messages

5 out of 8 participants experienced experienced confusion when a successful file upload triggered a false error message.

  • The issue: The file uploaded with success confirmation, but the system displayed an error state, causing users to doubt their success and attempt redundant uploads.

  • The impact: This broke user confidence and eroded trust in the system's feedback.

Misleading Error Message on File Upload

RECOMMENDATIONS

  • Fix the underlying bug causing false error messages on successful uploads


  • Implement distinct visual states for success, warning, error, and informational messages (currently, error and warning states are visually identical)

IMPACT

STUDY REPORT

Our research culminated in a readout to the Microsoft Foundry Knowledge team. Key outcomes:

  • The guardrail blocker (Finding 1) was flagged as a P0 fix: the team confirmed it would be addressed in their next sprint cycle, with automatic default guardrail assignment during agent creation.

  • The landing page redesign shipped within 6 weeks of our presentation. The updated page rebalances the CTA hierarchy, making "Start Building" the dominant action. (See before/after screenshots below.)

  • 3 of our 5 recommendations were prioritized for the team's Q2 2026 roadmap: guardrail auto-assignment, terminology audit, and post-creation guided flow.

  • The terminology audit is currently in progress, with the team evaluating renaming "Knowledge" and "Tools" based on our competitive benchmarking recommendations.

Before
After
Charlotte Truong Portfolio

REFLECTION

What I Learn

The biggest lesson was the importance of adaptability in moderated research. The guardrail blocker wasn't something I or Foundry team anticipated, it surfaced in our first session and caused 100% task failure at step one.


I had to make a real-time methodological decision: do I let participants struggle indefinitely (preserving ecological validity but losing data on the rest of the workflow), or do I intervene? I chose a hybrid approach. I observed and recorded 3–5 minutes of each participant's natural struggle to capture the severity and emotional impact of the blocker, then provided a workaround to unblock the remaining tasks. This ensured we documented the critical failure without sacrificing the other four findings.

What I’d Do Different

Add a structured debrief: Some of the qualitative data came from off-script comments after tasks were completed. Adding a dedicated 5-minute debrief with open-ended questions ("What one thing would you change?") would systematically capture these reflections instead of relying on spontaneous moments.

MADE WITH LOVE & MATCHA 🍵

Charlotte Truong

Product Designer

WORK

ABOUT

Resume

Charlotte Truong

Product Designer

WORK

ABOUT

Resume

Charlotte Truong

Product Designer

WORK

ABOUT

Resume