Recently I was involved in the launch of a multi-year project. It was a replacement of a company’s core internal application; a Transportation Management System(TMS for short), an application that powered an entire company; and, had daily usage from 99.99% of the 800 person company.
This was a lengthy project and it involved not just one set of stakeholders, but all stakeholders from all departments of the organization: Sales, Operations, Finance; and, within each of those high level departments were multiple teams, stakeholders and initiatives. All in all, there were around 72 PRD’s (Project Requirement Documents) that accounted for the scope of the features involved with this project.
There were many lessons learned in this project; however, this article is going to focus on the post-launch feedback loop.
For some context - this was a replacement of the existing TMS and, aside from a lot of strong architectural improvements, this also involved a complete and total redesign to the application. A lot of effort was made, pre-launch, to introduce the application and the changes to as much of the company as possible; however, once go-live happened, it became clear that a lot of people did not go through training.
So the story goes like this - you spend multiple years building an application. You spend a good portion of time validating, but then it goes live and you run into the real world. Assumptions made, desired business improvements proposed, none of it matters once you get to the reality of launch.
Go-live introduced a new phase of feedback that was exciting and perhaps the most effective feedback loop I have been involved in; and, this is what I most wanted to share.
We launched with two feedback channels. I believe getting valid user feedback is one of the most challenging aspects of the job. Noone looks at a website until it’s launched, is my own personal experience.
So the challenges are:
Getting User Feedback
- A feedback widget attached to the application. A user can click on a button to open a form and provide feedback. That feedback captures the logged-in user, the url they were on. The feedback widget was stored in the application itself; and, it was forwarded to a separate system (a Jira project) for review.
- A chat channel that started with an initial champions group and the product team. This chat quickly grew to 250 users and it morphed into a mix of questions, defects, and product requests.
Up until this moment, getting actual user feedback was tough. Until we had started ‘the champions group’, we didn’t much feedback, at all. It is a truism that noone really looks at a website design until they actually have to use it, and this was no different.
Now, however, faced with the hard reality that this was, indeed, the new application people used to do their job, now we actually got feedback, en masse. We received 1000 feedback notes within the first week and the champions chat grew from 45 users to 250 users, as we reached the maximum cap of users allowed.
We now had feedback and the challenge in front of us was a number of items.
Managing User Feedback
How to manage this volume - how to determine what to move on, what to ignore, etc. This is more than just add the feedback to some spreadhsheet and manually review. We got 1000+ feedback items. We needed to review the feedback, collate it for commonality, determine what was ‘nice to have’ and what was truly critical. We also needed to know what was already fixed, what was getting fixed, and what would not be fixed.
Communicating to Leadership
This was a big launch for the company. This project had been a 3.5 year project and the success of it was critical to the business. Communicating UP to leadership on a number of items (how was it going, what are we working on, what’s next) was critical on a day-to-day cadence. The biggest question was, indeed, “how is it going”? Our user base was quite vocal; however, even important than that, this was our primary application that powered the business. “How is it going” is what mattered
Communicate back to the users
How to communicate to the users, to the feedback channel, so they know their feedback was being heard and they knew what was being prioritized.
How did we manage this? We hooked up claude code to the submitted feed, and the chat history, along with our project management tool (Jira) and github. We were able to find commonalities in the feedback, to demonstrate what the users were asking for and we were able to prioritize, not just by volume but by criticality. We took that and compared it against jira to identify what was already captured, what was in progress, and what had been released.
And with this, we could communicate to leadership not only what had been delivered and what was being prioritized, but also how the rollout was being experienced: where users were frustrated, where confidence was improving, and whether the conversation remained constructive.
We ran this feedback loop for the first five weeks into the launch. During that time we deployed close to 800 different stories and defects and the feedback mechanism described was integral to ensuring we focused on the right work.