A reflection of my biggest failure in Product Design

Foreword: In 2014, Joshua Taylor led his design team to completely re-brand Penultimate, a digital handwriting app that Evernote had acquired. He wrote “My Biggest Failure” as a reflection of the work he did, where it went wrong and what he has learned.

I’ve made a lot of mistakes I guess—both personally and professionally.

This is the story of the biggest design mistake I’ve made. It doesn’t have a big happy ending, but that’s fine.

The Launch

On 10.29.2014, early in the morning, we launched Penultimate 6. It was a major, ground-up redesign — and not one part of the product was left untouched. These types of redesigns are tricky at best. You know some people will be unhappy, but the hope is that the new changes will improve the product for a solid core of users and open it up to many more.

We spent a lot of time weighing the idea of even starting the redesign. Ultimately, it was a yes. The app had reached as far as it would go in its current form. Evernote had bought Penultimate a few years prior when it was top on the charts of paid iPad apps. It was rooted in original iOS skeumorphic metaphors that severely constrained the new features we could add, as well as how the data in Penultimate could be used in the Evernote ecosystem.

There’s only so much you can do with an actual physical notebook, was what we thought about Penultimate.

After months of research, surveys, exploration, user testing, beta testing, iteration and playing with all these versions on our iPads, we decided to launch publicly.

The realization — a cacophony of criticism

Within a few hours after the launch, we had promotion from Apple, millions of people updating — and a few grumbling users. By day two, we had a growing list of serious complaints. By day three I wanted to crawl in a hole and hide from the world.

Making big changes to an app always comes with a bit of resistance. That’s to be expected. Generally speaking, people don’t like change — especially to something they love. I was ready for the complaints because they are necessary sometimes in order to move forward. What I wasn’t ready for was how accurate and insightful the criticisms would be.

I wasn’t ready for how accurate and insightful those criticisms would be. Image courtesy of author.

What went wrong

Testing & Segmentation

Showing designs to users early and often is an extremely important part of the design process for me. Penultimate was no exception.

However, I made one key mistake. Segmentation. Evernote has over 150 million users, and Penultimate users are a subset of those users. We included a lot of Evernote users in our user studies— even those that did not, and would not, use a handwriting app on their iPad—assuming that they would serve as a good proxy for Penultimate. This was a huge oversight.

False Assumptions

We thought they had a lot in common, but it turns out that most people who use an iPad for handwriting (Penultimate users) are in a very different mindset than someone who is on the go and typing into their phone (Evernote users). This seems obvious in hindsight. Our responses from testing were largely positive.

I also assumed that the skeumorphic metaphor of a notebook was dated and no longer desirable — that users would prefer the shiny new UI paradigms afforded by iOS7 (i.e. “flat design”). This was hugely false. The metaphor of a physical page was not antiquated at all. Paper is not a thing of the past. In fact, those metaphors are powerful in their ability to help users with a sense of space and navigation inside of the app.

Fixed Deadlines

We were working with a physical product — the Evernote edition Jot Script Stylus. Physical products have much longer timelines and don’t have the flexibility of release dates like software does.

I made promises that we would be ready to launch on a certain day. For many reasons, we simply weren’t ready. Even so, I felt the pressure to meet the dates we had set, and I allowed us to ship the product before it was at an acceptable quality level, and before it had undergone further user testing. This was silly. External timelines are the enemy. Other companies have their own objectives and should never dictate the quality of your app. #lessonlearned

Communication and Auto-updates

For many users, iOS automatically updates their apps. I knew this. Somehow though, I didn’t take into account the gravity of the fact that a user would open up their app one day and it would be completely different. The worst story I heard: A university student was taking notes in class, switched apps to look something up, and returned to the app finding that everything had disappeared after one auto-update. I don’t think the person would use Penultimate ever again.

For all of the changes we made in Penultimate, communication to the existing users was lacking. We probably lost a lot of people just because they didn’t feel like having to learn a new app.

An Evernote notebook in Penultimate 6. Image courtesy of author.

I’m sorry

This was a project I put a lot of my heart into. I led the product direction, the product management, and the majority of the design. It was painful listening to thousands of pieces of negative feedback — most of which were incredibly true. It hurts when you realize that your mistakes were harmful to someone. And when that someone is backed by millions of similar voices, it is borderline unbearable.

I’m sorry that I didn’t take the chance to make this app what it could have been, and I’m even more sorry that my decisions had the impact that they did.

Triage

We immediately took the feedback to heart—along with their pain—and started making improvements. I reached out to people on the forums to ask them what we could do better and then sent them prototypes of what we planned to do. We wanted to get it right next time. We reintroduced pages, we fixed bugs, we improved zooming, we scrambled to get in fix after fix. We apologized.

But in some ways, it was too late. Users felt betrayed. And because of that betrayal, our bar had been raised and it was even harder to meet those new and higher expectations.

Where do we go from here

I’ve probably learnt a lifetime of lessons from my mistakes on Penultimate. For now, I hope there’s value in simply admitting that I made mistakes. It’s too easy to talk about wins after a failure. But this is where I share about the failure itself, and pain that comes with it.

Despite the mistakes made in Penultimate’s redesign, I still think it has some amazing features. In fact, it remains one of my favourite pieces of work in my portfolio.

Yes, I’ve learned a lot. But let’s call this what it was. It was my biggest failure.

One final thought

This is in no way meant to be a negative reflection on Evernote or anyone else on the team. I have a ton of respect for everyone I worked with on this. I decided to leave Evernote 6 months ago to help startups launch new products (and hopefully avoid a lot of these mistakes). Evernote has continued to make improvements. The app is now back up to 3.5 stars and heading in the right direction due to their hard work.


Joshua Taylor spent his career designing early ideas – Sometimes that means getting a new product off the ground for a brand new startup, and other times he works in house at bigger companies, leading teams to launch or redesign major new initiatives. He has also led highly talented design teams at Credit Karma and Evernote.

Disclaimer: All content is written by and reflects the personal perspective of the writer. If you’d like to contribute, you can apply here

Leave a Reply

Your email address will not be published. Required fields are marked *

Discover Oasis

Oasis is meant to bring human stories to life and serve as a focused, borderless platform for readers to gather, learn, and interact.
It is a brainchild of our sister website, KrASIA.

Join Our Community

Contact Us

Find Us on Social