ProvantageX -
Lean Agile UX Process
& Post Buy Stewardship

What is ProvantageX?

PVX is an end-to-end B2B SaaS web-based platform that allows the purchase and sales of local broadcast and cable tv advertisement spots.

TV advertisement is a multi-billion dollar industry that was lingering in the past, with most buyers and sellers closing deals with phone call, excel sheets, emails and print outs. This gave PVX a unique opportunity to develop a program with an intelligent self-learning algorithm that would optimize for the best spots that would give the buyer’s clients the most target demographic impressions, with reduced cost while still hitting the client’s daypart goals and avoids hit list shows.

Some of My project Case Studies from PVX

Cable Project - Cable integration into Buy Specification (Case Study Coming Soon)
Admin Project - My Contacts (Case Study Coming Soon)
Admin Project - DSP User Permissions (Case Study Coming Soon)
Adjust Tool 2.0 - Mass Inventory / Buyline adjustment (Case Study Coming Soon)
Bulk Program Adjustment Tool (Case Study Coming Soon)
XML Export & compatibility with External Systems (Case Study Coming Soon)
Data Display Options Improvement (Case Study Coming Soon)

PVX TV Media Stewardship

The Kick Off - Understanding the Industry

I joined PVX without any media knowledge, so I spent most of my first couple months learning media terminologies, how media math works, and to accomplish that I had to read the documentation on the existing system from PVX and interview internal Business Analysts and external media buyers, planners and assistants from agencies, and seller & media planners from stations and rep firms.

When it comes to buy stewardship, only the sales executives and assistants are the primary users on the supply side, and the buyers and assistants are the target consumers from the demand side. So those two user archetypes and workflows will be the primary focus in stewardship.

Insights & Discovery

Other Considerations

1. Quantity of Data - traditional Media Buyers are always analyzing a massive amount of data. When the buyers are analyzing makegood proposals from the sellers, they usually have multiple spreadsheets open analyzing the active buy total value, spots, impressions.
   a) Display Options - Because there are a massive amount of quantitive data such as airtime ingestion time, Market of the buy, Station, Daypart, acceptable days of the buy spots, Time frame of the daypart (each buyspec has different time range definitions), TV program, PVX Program (Part of PVX’s benefit of intelligent program matching using self learning AI system), ad length, cost, guaranteed rating (rating purchased), actual rating of the buy Weekly spots. These information also cascade from the higher buyspec performance level, to individual station buy for the buyspec, to individual buylines, to airtime, there a lot of data that needs to be displayed and considered every step of the way.
   b) Sortability - Because each buyer and seller would handle multiple buyspecs at a given quarter, and each buy could have hundreds to thousands of buylines, it’s important for them to be able to quickly sort through the data efficiently.

2. Demographic Data - In TV advertisement, agencies(buyers) and stations(sellers) are usually subscribed to Nielsen Data or Comscore data for demographic information. However there are cases where a station are not subscribed to neither or both. It is illegal to display Comscore and Nielsen data at the same time. It is also illegal for a station to send over Nielsen or Comscore data to an agency that isn’t. PVX need to avoid legal issues.

3. Broadcast TV vs. Cable - Thus far, all the buyers and sellers I have interviewed are local broadcast TV buyers. PVX currently does not support Cable buys but cable buy workflow should be considered in order for the integration to be an easier process in the future.

4. Data Synchronization - There are two aspects of data synchronization.
   a) Internal Data Synchronization - PVX needs the data to sync from demand side platform and supply side platform, therefore we want the buys to be locked if the supplier proposed makegoods or if the demand side made any changes to the buy. Both parties needs to accept the incoming changes before they can work on the active buy.
   b) External Data Synchronization - PVX needs to sync data to external system ePort by Strata for agencies for as a client billing and auditing system.

5. Standardized Terminology & Format Consistency - In the past 5 years before Product Team joined PVX, it was built without user research and usability-testing. PVX needs to use industry standard language and terminologies the user can understand. For example, PVX uses the term “Advertiser” and “Client” interchangeably through out the system. However upon investigation, we found majority of the traditional media world adheres to a “Client/Product/Estimate” format.

All these things and MORE became part of the overall scope and issues PVX needs to cover to truly be functional and have a competitive edge over traditional methods in the stewardship workflow. Those items became items to address in the product team UX backlog.
But hold on, that’s too much to build at once! How did we decide what’s the MVP (Minimum Viable Product), and what we will add to the UX backlog for future iterations?

User Story Mapping
Planning out MVP & Future Phases

Innovation involves risk. To mitigate risks, the product team conducted workshop meetings with internal stakeholders separating assumptions from facts. A user-story map takes the perspective of the product. It aims to guide the planning and implementation of features and functionality to solve primary users’ essential needs by prioritizing what we know and testing our assumptions and features that we THINK will benefit the partner companies.

Part of our challenge was the time constraint due to upper management’s promise for certain milestone goals for each quarter, we had to figure out how to build the essential workflow and areas where we can bring the most time and cost-saving benefits to the primary users (Agency buyers in this case). We also needed to bring features to attract more suppliers to the platform and they are our secondary external stakeholders.

Partnering with PVX scrum master Seung and two members of the data analysts and our product owner, we sorted items above into affinity maps and created overarching themes. The themes dictate what we must build. They are put into a system-based workflow: User Story Map. The MVP are built on proven facts, what the buyers and sellers NEED. The user stories that contain assumptions need to be tested before moving them to epics. Our UX backlog is separate from the dev backlog that Seung also manages.

User Stories
Turning Large scale project into achievable goals

Each step, or epic is broken down by user stories, identifying users fo the feature, motivations and benefits.
The user stories are completed with acceptance criteria from our backlog refinement sessions. New acceptance criteria are sometimes added after my design spikes that pushes the product forward.

I would create wireframe and prototype designs to fulfill the acceptance criteria from this point. In some design sprints, product owner Amy and data analysts and I also brainstormed and created additional use cases and solved for edgecases for each user story

De-risking Assumption with Experimentations

Each item that is not proven as fact is tested at the same time.
During our workshop meetings with core, stakeholders cost-saving or benefit-driven ideas are generated. It is the product team’s responsibility to document all ideas and create experiments to turn those assumptions into facts to minimize the risk of wasting company time and effort.

The majority of PVX’s benefits are based on AI automation. However, it takes a long time to develop intelligent and efficient AI and it’s risky to build an unwanted product based on a hunch. We often conduct experiments soliciting feedback from our clients. Beyond that, we also use the Wizard of Oz workshop to test our prototypes. We create a script that outlines responses and actions of the system and our participants will act as the user types. They will initiate a task and our role is to capture their problem and frustrations to capture gaps in my design and see what can be improved upon.

My role during workshops is usually the facilitator while the analysts take complete notes on the transcript and write out notable results supporting the artifacts generated from the exercises.

Backlog and Kanban Board

Story mapping and user stories are for project planning. Kanban board is there for project tracking. PVX has 2 product teams with a product owner working with both teams, 2 data analysts and 1 UXer handling the entire research and design process.

The Design Process

This is the foundation of all wireframes and prototypes. It is a flexible platform architecture that gives suppliers the freedom to send Airtimes to agency users first with or without the initial easy-matching process. They also have the option to continue with a makegood proposal before buyer response. However to ensure internal data synchronization, when the seller is editing makegood proposals the buy becomes locked for the buyers until the seller proposes the buy. The buyer could still work on other buyspec or their other buys for the same buyspec in the meantime. When the seller updates the active buy, the buy is also locked for the station. until the seller send over their updates and the seller has to confirm or reject first before starting their proposal workflow.

The prototype was built using Sketch exported into InVision for PVX client and partners testing and review before development. But what do all these steps mean and how did we get there?

Workflow Mapping

From the process map and story map, the product team identified the proposal review process as the most benefit-driven part of the platform. The Supply-side can do their work externally and upload XML for buyers to address as part of the MVP process. The buyers are also the primary users of the platform. The entire workflow took nearly 9 months of an iterative process to create and improve. For brevity only the makegood prosal process is showcased above. This step is usually done with wireframes that represent the concept but not the final design.

Usecase and annotation

There are a lot of thoughts and consideration that goes into every part of each screen. Internal stakeholders such as backend dev would give input on feasibility that we aren’t aware of. Upper management would give feedback on various aspects of the design. Sometimes I would create different options as a part of the design spike process and the entire team would vet designs under the guidance of my UX principles. Some designs are stress-tested against our user research experiments as part of the UX scrum cycle to solve user stories.

However, not all developers who will work on implementation are part of the scrum team. So it’s important for us to not only how each component functions but also why certain decisions are made. It becomes part of the artifact that can be revisited for QA testing and future iterations of the project.

These use cases were analyzed for this epic

To fulfill the acceptance criteria, we thought about these use cases the buyer experiences during our design workshops. They helped us solidify the features we want to include in the product.

Use case #1

WAGA Station with Nielsen Subscription sent Makegood Proposal. Agency A buyer Tom needs to address a Multi-quarter buy with 3 proposals from WAGA.

Use case #1.2
Tom decides to unselect all active proposals.

Use case #1.3
Tom decides to address Proposals 1 and 2 at the same time.

Use case #1.4
Tom decides to accept proposal 1

Usecase #1.4.1
Tom decides to accept Proposal 1. However, he wants his WAGA sales executive to know that he is not happy that it missed the time frame defined by buyspec daypart definition by more than 5 minutes. He will not accept another proposal like this in the future.

Use case #1.5
Tom decides to reject proposal 1. WAGA sales Executive working on this client for Tom wants to know why Tom rejected the proposal, so he can work on a makegood Tom will accept.

Use case #2

WKTB Station without Nielsen Subscription sent Makegood Proposal. Agency A buyer Tom needs to address a single-quarter buy with 1 proposal from WKTB

Usecase #3

WGCL Station with Nielsen Subscription doesn’t have any active proposals to address. The buyer Tom comes to this screen to see all their active buylines for the buy.

Interaction Design

In most cases, Figma is a great tool for interaction and visual design. The auto-layout and constraints make designing with scaling easy and painless.

However, PVX is a complex b2b software that requires complex logic-based interaction. I was required to build mathematical calculations that dynamically change prototype data based on user input and/or selection. I was able to accomplish this using Axure’s local variables, global variable functionalities, and conditional interactions.
Figma alone cannot fulfill the needs for PVX development. Therefore I opt out for Axure RP10 as the primary wireframe and interaction design tool.

Style Guide & Design Documentation

Sometimes the design for a certain part of the system does not conform with standard system behavior. For example, part of agency buyer’s painpoint was that they can not see a certain buyline, buy, and summary information at the same time, which is necessary for them to make an informed decision. Prior to the update, they have to scroll to a particular section, take a screenshot, then scroll to another section to find a particular line and take another screenshot, then compare screenshots by opening them next to each other on PC.
This is a persistent issue for many parts of the system.

To solve this particular issue, I created this new system standard of displaying multiple stand-alone scrollable tables that scales situationally on the page so the maximum amount of each table can be displayed at once taking advantage of the screen real-estate. However this behavior cannot be easily documented with prototyping and workflow mapping alone, so I had to create an appendix to the style guide with rules and formulas for how the tables interact with each other on-screen with preset conditions and calculations.

This piece of documentation has been used for many QA tickets on dev scrum cycles.

Sprint Demos

Sprint demoing is part of UX ceremonies. This doesn’t necessarily always happen at the end of our UX sprint cycles, sometimes we demo an idea or wireframe prototype to gather stakeholder feedback and check dev feasibility before we investigate any further. However, prior to project hand-off for dev sprint planning, we always demo the prototypes to all internal stakeholders first.

During the demo process, the team analyst and I would work together to work on the intro for the meeting. How we approach the script for the meetings varies based on the target audience. For upper management we would focus on various use cases and what we designed to solve for particular issues and talk about alternative designs we had and why those didn’t work if necessary.

For dev meetings, we tend to focus more on the functionality and interaction of the system.
My role is usually going through the presentation and responding to stakeholder questions while the analysts would take notes and record the meeting transcripts. The product owner Amy usually acts as the timekeeper for us, making sure we don’t go overtime and keeping us on track if the conversation deviates from the original problem or opportunity statement.

Handoff Documentations QA & UAT Tickets

Handoff

Other than uploading data documentation along with UX artifacts and prototype links the product team usually writes a hand-off email to the dev team with all the stakeholders on the email write-up. The dev team would then turn our workflow and break it down into dev sprint cycles and assign tickets to dev team members.

QA & UAT

Due to the complexity of our projects, each designer and analyst would stay with their respective project for the QA process & usability testing to aid the process. The QA team would also reach out if they have any questions for us.
Any technical or UI bug that is found is reported on Jira with priority tags, screenshots, and descriptions. The desired result and original artifacts are linked to tickets as well.

We would also perform usability testing for scripted use by our internal QA team and behavioral testing against our external stakeholders.

Our internal QA team is an advanced user of our system and knows how it’s supposed to work. They also read update notes written by our QA director Abigail. We test against them for time and cost-saving metrics. This can only be tested against the QA version of the system with inventory data that are bought by our test-buyers via the competitive process (price negotiation).

Against our external stakeholders our testing are more about observing their behavior when asked to complete a certain task. We want to see if they use the updated features and if so, how they would use it, any part they would pause and look confused. In doing so, we can make sure the system is natural to use and learnable by the users.

Retrospect & Random Thoughts

Early stakeholder onboarding

If a stakeholder is part of the decision-making process, onboard them on ideas and solutions AS SOON AS POSSIBLE. Sometimes crafting the perfect solution that solves for all use cases and edge cases can be a painful grinding process. It can save the product team so much time to just present and introduce unpolished rough concepts and get some review and feedback first. We introduced dev managers to the UX scrum team in some of the later projects because during this project, we have often thought that we have crafted the perfect solution, but then had to change direction later due to dev feasibility issues. Sometimes the data team or business team would give us additional insights that would generate new use cases that would cascade into design direction change for the project as a whole.

There isn’t a perfect solution that makes everyone happy

Sometimes there just isn’t a perfect answer that solves all use cases. The UX workshops I have learned from the NN/g group were extremely valuable for this specific reason. Stakeholder mapping helps us identify who would have a higher interest and influence on product benefits and on a tight time constraint, we would work on those user stories first.

Sometimes a feature has a high impact however the complexity of the project makes it not worth prioritizing, and I would propose an incremental design that satisfies the acceptance criteria but without automation to reduce R&D on the feature.

Diversify the testing sample

There isn’t a single “Agency Buyer”. Every agency has a different structure. OMG Media Group takes a market-first approach and each buyer works for different clients in their market. Whereas some other agencies who work with PVX like Trilia - IPG takes a client-first approach, each buyer would serve their clients across all markets. They are there to make sure the campaign runs smoothly across the country. It was important for us to understand how all agencies work and design a system that accommodates all persona types.

Timebox is useful for more than just workshops

Sometimes putting on a time constraint is helpful, especially with the dual-track scrum model. The product team is full of intelligent and talented individuals but sometimes we are guilty of over-analyzing a situation, especially because there are so many internal and external stakeholders involved and billions of dollars on the line.

UX is a Team Sport

Collaboration is essential when it comes to idea generation. There are times when I would draft design solutions on my own to solve problems to achieve a goal (design spike). But sometimes a data analyst or product owner could provide better concepts and ideas during design review meetings by building upon a concept.

It’s important to collaborate often and be open to others’ ideas, feedback and criticisms.
It’s also important to compromise when it comes to dev feasibility. Build what must be done first and add additional beneficial features later.

Thank you for reading!