Jason Chen is

  Art     Design     Photography     About

Smart Notes, Product Design, 2018

Smart Notes is a mobile app using machine learning to document and analyze a meeting or conversation in order to provide suggested reference, organized key points, transcripts, and analysis.


An efficient way of collecting and analyzing information from meetings.

Using Smart Notes can help users to effortlessly collect valuable information from meetings, for example, business meeting and school critique. By utilizing machine learning, the algorithm will extract and analyze conversation among different respondents to generate suggested reference, organized key points, transcripts, and analysis.


People often miss lots of information during a meeting because they have to concentrate, take notes, give feedback at the same time.


Introduce a machine learning algorithm to assist people in documenting a meeting, also generate notes, references, and transcripts.


There are simply too many things to listen to and take notes in a meeting.

Every time during a meeting, we always get so many ideas, references, and suggestions which are very hard to catch up in a short amount of time. So we usually miss lots of information while taking notes and listening simultaneously. Record the whole meeting could be a way to solve this problem, but it could be burdensome to listen to the whole conversation again.

User Research

Meeting sometimes starts with joy but ends up with frustration.

Students who need to have critiques frequently are one of my main users. Therefore I executed 8 interviews with my classmates by inquiring them the experience of meetings, starting from prepare to take notes to organize feedbacks and key points. This identified users’ pain points and gain comprehension of their experience.
From the interviews, I noticed that most people shared similar experiences when during school critiques they receive unsure names or terms which needed to be looked up afterward. So the meeting sometimes starts with joy but ends up with frustration.   

User Journey Map

Taking the outcome from interviews, I created this user journey map, which includes the process from preparation to meeting to review notes. Also the experience level during the whole journey. This map gives me the context and the guidance to the solution.

Competitive Analysis

From the user research and journey map, I discovered people mostly use Evernote and Bear for taking notes. Therefore, I further analyze those apps to identify their success and opportunities.


Based on research, currently, we can’t find a tool that can assist people effortlessly and efficiently document a meeting. Also, we find a specific need that people do struggle with concentrating and taking notes simultaneously. The question is how can we improve the experience? Our approach is to utilize machine learning to introduce personal assistance.

How can I design
with machine learning?

After reading the guideline from Human-Centered Machine Learning by Josh Lovejoy and Jess Holbrook. I saw these interesting example exercises they walk through while they are trying to address with ML. I tried to answer those use case scenarios:


Describe the way a theoretical human “expert” might perform the task today.

1. to differentiate small tick-talk/ joke/ sarcasm within a conversation
2. recognize voices from different respondents
3. look up the dictionary or google
4. archive/ sort texts and data

If your human expert were to perform this task, how would you respond to them so they improved for the next time?

Users will be asked to delete unrelated keyword and highlight relevant keyword before the computer starts to generate the report. This way, the algorithm will start to recognize what is important and what is not.

At this point, we have to step back and consider does it require ML, is it a meaningful enhancement by ML, and which do not benefit from ML or are even degraded by it.
A perfect example would be Gmail’s phrase detection. Gmail looks for phrases including words like “attachment” and “attached” to pop a reminder when you may have forgotten an attachment. The ML definitely works great here.

Giving feedbacks helps machine to learn.

Low-Fi Prototyping

Wireframes were created for initial user testing. In this version, there were only two tabs, Meeting and Profile. After a testing session with 10 respondents, I discovered some major problems in this first version.

Iteration 2 was rapidly created for another round of user test. Since Smart Notes do not only focus on documenting notes but also, more importantly, collecting and showing the contents, so I bring both analysis and saved page to the tab level.

Iterative Prototyping

Further investigation into the interaction and user flow of each page.

To help algorithm learns, drag to delete or enlarge to accentuate.

Users can delete unrelated keywords or enlarge important keywords to help the algorithm to recognize the relevance of reference.

Provide a preview statistic after a meeting.

After done recording the meeting, it will jump into the report page. Instead of directly showing the result and toggling between contents, it makes more sense to show a preview page about the overall statistic first. Then users can tap on them to see details in the report.


Why not directly show the content in full screen?

When users reviewed their reference, it’s unnecessary to show a pop-up screen almost as big as the screen. So I turned it into a slide-in full-screen article.

More modern design and sleek animation.

The text size was increased and the respondent was highlighted with a red accent for better legibility. Also, I deleted the text box and added a smooth animation for better user experience.


Data Visualization

The analysis function is essentially a data visualization showing a user’s trajectory over a period of time. My approach is to use different colors to represent different domains. If the user wants to inspect a specific domain, they can simply tap on it, and it would be highlighted.

Hi-Fi Prototyping

Start a meeting and let the machine helps you take notes.

See the well-organized reports.

Review saved references, analysis, or export to other devices.

Interactive Prototype


This project progressed along with our discoveries while researching and testing. Our intention was to create an application that can help people experience and document meetings in a simple way, while also helping create a comprehensive report. The result allows more positive benefits to arise, like generating analysis to show user’s trajectory and exporting to pages. Our next step would be to look into different scenarios such as business meetings. Although our prototype focused on a specific community of design and technology student, we believe that this idea has the potential to grow in outside communities as well, providing an effortless meeting.

© 2021