Microsoft

Designing for AI, before AI was the phrase

The Microsoft Education products I worked on were AI-native before the industry had a word for it. Immersive Reader is a UX surface over Microsoft Cognitive Services: neural text-to-speech, neural machine translation, NLP for parts of speech and syllabification, and computer vision for the picture dictionary. OneNote Live Captions streams Microsoft Speech Services speech-to-text and Microsoft Translator neural machine translation into a class notebook in real time. Reading Assignments measures student reading fluency using speech recognition models against an expected text.

The UX problems were the same ones I work on now at Pulumi, in Governance and in policy remediation with Neo: how to make ML capabilities legible at a glance, how to design for model confidence and ambiguity, how to fail gracefully when the model is wrong, and how to keep an experience accessible when the underlying intelligence is doing a lot of invisible work. The substrate has changed in the years since (those were per-task neural models; today's are LLMs and agents), but the design discipline that surrounds them is continuous.

Originated the Microsoft Education assignments schema

In 2014, I was responsible for creating the assignments schema, the data model that became the foundation for Microsoft Classroom, then School Data Sync, then Microsoft Teams Assignments. The work originated under Microsoft Classroom and propagated forward as that product evolved into Teams.

• Achieved the transition of the Microsoft Classroom and School Data Sync SaaS APIs into Microsoft Teams Assignments by leading conceptual UX, design integration, and full-stack engineering on the migration.

• Class Notebook in Microsoft Classroom. As Senior UX Designer in 2016, I designed the OneNote Class Notebook flows inside Classroom: setting up a notebook for a class, distributing pages and content to students, and creating sections. For Classroom Assignments, I owned the design for managing class and notebook membership, retiring the separate OneNote assignment section, and locking the collaboration space.

• Secured funding for Microsoft Education and successfully drove formal product group creation.

• Coordinated longitudinal research studies whose findings shipped as extensible features, work that originated in Microsoft Classroom and propagated into Microsoft Teams Assignments and OneNote class notebooks.

Innovated Immersive Reader, zero to 46M MAU

I innovated the Immersive Reader web application concept and took the app from zero users to a fully formed reality during my Design Developer 2 tenure, then supported its scale to 46M monthly active users across multiple platforms.

• Created the React prototype that scaled directly to production, and improved its portability so any web experience could embed it.

• Scaled for Microsoft Office. We scaled the service and the UX to support multiple products with one stateless app and a portable React and TypeScript front end built with Fluent UI. Every Microsoft app that offers Immersive Reader, except Edge, runs this shared reader: Word, OneNote, Outlook, Excel, Teams, Forms, Lens, Reading Progress, Reading Coach, LinkedIn and Minecraft Education (Microsoft's list).

• Minecraft Education, the widest reach for kids. I did the Minecraft Education integration, launched in 2019. Students press a key to have signs, character dialogue and boards in the game read aloud, translated, or reformatted, with a picture dictionary of Minecraft imagery. Of all the integrations, it probably did the most for children.

• Preferences that follow the reader. As a design engineer, I implemented user preferences that persist the preferred reading mode across all Office apps. Set it up once, and the reader opens the same way in every app.

• Helped shape the public Immersive Reader SDK, the JavaScript/TypeScript embed library that wraps the Azure Cognitive Services token exchange and lets any web experience launch the reader behind a few lines of code. Full-stack work: the JS/TS client API, the documented integration pattern with the backend token service, and the surface those choices created for developers integrating it.

Opening Immersive Reader to every app

I worked to make Immersive Reader something anyone could embed in their own apps, not only Microsoft. I'm the most active person in the SDK's public repository by commits.

• Quickstarts for every platform. I wrote the sample apps developers start from: Python, Java, Android in Kotlin and Java, and iOS in Swift, each showing the secure token exchange with secrets kept out of the code.
• Ready for general availability. I moved the samples to SDK 1.1 for its August 2020 launch as an Azure Cognitive Service, and kept the SDK healthy after it: API fixes, validation, and security updates.
• Impact. At general availability, more than 23 million people a month were using Immersive Reader, and usage grew 560 percent between February and May 2020 as schools moved to remote learning. Microsoft now lists more than 40 partners that embed it, including Canvas, Schoology, PowerSchool, Nearpod, BrainPOP and Code.org, and says integration can be done in less than a day. The SDK still sees about 350,000 npm downloads a year.
• Feedback. Chicago Public Schools, on the launch: "Immersive Reader is a crucial support for CPS students and families."

Sources: Azure blog, August 2020 · Immersive Reader partners · npm

• Lead UX/UI prototyping for Reading Assignments and Immersive Reader, focusing on process, production, and engineering.

• Contributed critical code to three winning Microsoft Hackathon projects (best overall, best in category).

OneNote web accessibility

Drove success within the architecture of OneNote web application accessibility features, achieving SLA attainment of Level-C accessibility standards, Microsoft's strictest internal accessibility scorecard.

Other shipped work

• Fresh Paint on Windows 10: created code for Fresh Paint's final release, an application still available on Windows 10 devices many years later (full Windows 8 case study here).

• Office Mix Mix Gallery: collaborated with design and engineering to implement Mix Gallery, enabling unlimited user sharing through front-end Office Mix code.

• Microsoft Classroom & School Data Sync: transitioned to leading engineering and front-end development for Microsoft Classroom and School Data Sync during the Design Developer 2 era.

• On-call rotation: managed engineer rotation and live-site incident mitigation, including resolution tasks, telemetry implementation, and continuous-delivery-focused DevOps.

How I worked

• Drove the cadence of quick-cadence executive presentations representing Microsoft Education product vision, iterating quickly as products evolved and keeping internal stakeholders informed of UX progress.

• Operated at the seam between design and engineering across three role evolutions: Senior User Experience Designer → Design Developer 2 → Design Engineer 2.