Echo Frames LPM x Device Arbitration Evaluative Study

Worked on the third generation of Amazon's Echo Frames, smart glasses designed for on-the-go Alexa voice assistance and listening.

Problem 1: Device Arbitration: Users reported frustration when saying the Alexa Wake Word, as it would unintentionally trigger other Echo devices instead of the Echo Frames. This caused dissatisfaction because users wanted responses specifically from the Frames.

Problem 2: Battery Life Expectations: Beta test surveys revealed low customer satisfaction (CSAT) scores regarding the Frames' battery life. Engineers were developing a new Low Power Mode (LPM) feature, which had not yet been tested by users. The research sought to understand user expectations for battery life and how they envisioned LPM functionality, including potential trade-offs like disabling the microphone when LPM activated. 

The research goal was to prioritize problems for improvement before and post-launch, with the product scheduled to launch in the Fall.

Investigate how device arbitration issues impact user experiences and identify potential solutions to improve satisfaction.

Explore user expectations around battery life and gather feedback on the proposed Low Power Mode (LPM) feature, including its trade-offs (e.g., disabling the microphone).

Provide actionable insights to guide design and engineering priorities for pre-launch fixes and post-launch refinements.

Method Selection: Conducted qualitative interviews with beta testers who reported dissatisfaction with device arbitration and battery performance.

Why Interviews? Surveys provided general feedback, but interviews allowed for deeper exploration of user experiences, expectations, and reactions to trade-offs for device arbitration and LPM.

Process:

• Recruited beta testers who had reported device arbitration issues or low satisfaction with battery performance in earlier surveys.

• Designed and executed a semi-structured interview protocol focusing on scenarios of device arbitration, user needs for battery life, and potential use cases for LPM.

• Conducted remote interviews, transcribed responses, and performed qualitative thematic analysis.

Project Leadership: Led the research end-to-end, coordinating with stakeholders (Product Manager and UX Designer) to align on goals and study scope.

Research Design:

• Planned and ideated the study design based on stakeholder needs.

• Developed the interview protocol and refined it with feedback from the UXR team.

Participant Recruitment: Identified and recruited beta testers based on survey responses related to device arbitration and battery performance.

Execution:

• Conducted remote interviews, transcribed and analyzed data, and synthesized findings into actionable insights.

• Created a detailed report and a concise 2-minute slide deck to communicate findings.

Stakeholder Communication: Provided regular updates and ensured alignment on findings and implications.

Presentation & Follow-Up: Presented results at an org-wide meeting, which led to a detailed follow-up session with the engineering team to explore findings further and discuss next steps.

Device Arbitration:

• Users developed creative but cumbersome workarounds, such as cupping the microphone on the Frames or leaving the room to ensure Alexa responded from the intended device.

• Some users discovered they could change the Alexa Wake Word specifically for the Frames, but the setup process was unclear and poorly documented.

Battery Life Expectations:

• Users expected longer battery life for continuous use during activities like meetings or outdoor walks, expressing frustration with frequent recharging needs.

Low Power Mode (LPM):

• Users appreciated the idea of LPM but expressed concerns about its functionality, especially when the microphone was disabled during activation.

• Specific scenarios, such as LPM activating during meetings and cutting off the microphone unexpectedly, highlighted the need for more control over its activation.

• Preferences emerged for a manual toggle or customizable settings to control when and how LPM activates.

Actionable Insights:

• Provide an intuitive manual toggle for LPM activation directly on the Frames (e.g., a button press).

• Enhance documentation and onboarding to clarify the setup for LPM activation and custom Alexa Wake Word configuration to avoid device arbitration issues.

• Set realistic user expectations for battery life in marketing materials and user guides.

Impact:

• Following the presentation and follow-up discussions, the engineering team decided to allocate more time to refine the LPM user experience. The feature was scheduled as a post-launch update to address identified issues and incorporate user feedback.