How to Compare Problem Severity Across Customer Segments
To compare problem severity across customer segments, fix each segment by an observable trait before collecting data, then measure every segment with the same occurrence definition, the same observation window and the same unit. Compare medians and ranges for frequency and for the yearly cost of the current workaround, and count completed commitments per decision maker asked. This suits founders choosing a first segment; the main constraint is small samples.
The outcome and observation window
This guide is for founders at the idea-to-validation stage who hear about the same problem from more than one kind of customer and need to choose which group to pursue first. The output is a short comparison that says, for each segment, how often the problem happens, what the current workaround costs and how many decision makers completed a costly step, with each figure tagged by how it was obtained. If the choice is between different problems for one kind of customer, use Is Your Startup Solving a Problem People Will Pay to Fix? instead. For a sheet that covers a single account, use the Problem Validation Scorecard: Pain, Frequency, and Budget instead.
Severity here means three things measured separately: frequency, cost and commitment. Because the point is to compare segments, the measurement has to be identical in each one. Pew Research Center's guide to writing survey questions says that even small wording differences can substantially affect the answers people provide, that identical question wording should be used when the intention is to compare results to those from earlier surveys, and that the order in which questions are asked can influence how people respond. That guidance is about surveys repeated over time. Applied to a segment comparison, it means asking every segment the same questions in the same order, so that a difference between segments is not a difference in the questions.
Before measuring, write down four things:
- The segment rule. An observable trait that decides membership before any answer comes in, such as ownership type, billing model or number of locations. The U.S. Small Business Administration's market research guide advises: 'Be specific when you name your target market.' A segment that can only be recognized after the interview is a pattern found in the answers, not a rule.
- The unit of analysis. The Census Bureau's glossary for Statistics of U.S. Businesses defines an establishment as 'a single physical location at which business is conducted' and says a company 'may consist of one or more establishments.' Measure frequency and cost per location, and count commitment per decision maker, because the owner of several locations makes one decision.
- The occurrence definition. What counts as one occurrence of the problem, written so that two people would count the same event the same way.
- The observation window. The same calendar period for every segment, long enough to include the problem's normal cycle. If the problem is seasonal, comparing a segment measured in a busy month with another measured in a quiet month compares months, not segments.
Set the decision line now as well: the price you intend to charge, the margin the yearly cost has to clear and the costly step you will ask for. Writing it down before the data arrives keeps the comparison from bending toward the segment you already prefer.
Problem frequency: definitions and evidence
Count occurrences per location inside the shared window, from the account's own records where possible: a log, a timesheet, a ticket queue or a chore calendar. Tag each count said, seen, done or assumed, as in the Customer Interview Notes Template for Founders and keep the tag next to the number in every table.
- Normalize before comparing. Divide by locations, not by accounts, when segments differ in size. A chain of several sites will look like a heavy sufferer only because it has more sites.
- Report the median and the range. With a handful of locations per segment, a single summary hides too much. Show the middle value and the lowest and highest counts.
- Report the share from records. A segment whose counts mostly come from memory is less certain than one whose counts were seen, even when the medians match.
- Keep zero counts. A location that had no occurrences in the window belongs in the segment's figures. Dropping it makes the segment look worse than it is.
Frequency multiplies cost. It does not replace it. A segment can lead on frequency and still carry almost no cost, which is the pattern of a frequent inconvenience.
Cost of the current workaround: comparison and caveats
For each location, cost one occurrence of the current workaround, multiply by the count and scale the window to a year, with the scaling written down and tagged assumed. Use the same cost lines for every segment:
- Cash out. Repairs, fees, credits, contractors and supplies, checked against invoices.
- Paid hours. Staff time that would really be cut or moved to other paid work, at the account's own rate from payroll.
- Unpaid owner time. Hours the owner spends personally. Record them in hours and keep them out of the cash total unless the owner would otherwise pay someone to do the work.
- Lost revenue. Orders, bookings or customers lost because of the problem, from records.
Then compare each location's yearly cost with the intended price and the required margin, and summarize each segment by its median and range. Show the mean beside the median as a check. The measures of location section of the NIST/SEMATECH e-Handbook of Statistical Methods says that extreme values in the tails distort the mean and that 'these extreme values do not distort the median since the median is based on ranks.' A single large repair or one unusually busy location can lift a segment's mean past the margin while most of its locations sit below it. When the mean and the median point to different decisions, find the location that causes the gap and check that figure against its record.
Two caveats apply. Costs passed on to the account's own customers are not a cost to the account, even when they are real. And a one-off cost, such as a single large repair, should be shown apart from the recurring cost, with its chance of repeating tagged assumed.
The segment severity scorecard
The scorecard below is a problem severity scorecard specialized for measurement across segments. Fill one column per segment, use the same definitions in every column and keep the evidence tags next to the figures.
| Measure | Same definition for every segment | Report per segment | Watch for |
|---|---|---|---|
| Segment rule | An observable trait fixed before the first conversation | Locations and decision makers measured | Accounts moved between segments after their answers came in |
| Unit | Location for frequency and cost; decision maker for commitment | Both counts, side by side | One owner with several sites counted as several decisions |
| Frequency | Occurrences per location in the shared window | Median, range and share counted from records | Counts from memory; dropped zero counts |
| Cash cost | Yearly cost per location, with recurring and one-off costs shown apart | Median, range and the mean as a check | A single extreme value moving the mean |
| Unpaid time | Owner hours per location per year | Median hours, not converted to money | Owner time priced at an assumed rate |
| Commitment | The same costly step, price and deadline for every decision maker | Completed steps out of decision makers asked | No answer counted as anything but a no |
| Decision line | Price and required margin written down in advance | Whether the median clears the price and the margin | Moving the line after seeing the data |
A segment ranks ahead only when it leads on cost and commitment with figures that were seen or done. Leading on frequency alone does not move it ahead.
Willingness to commit money or time: scenarios and review triggers
Make the same request to every decision maker in every segment: the same step, the same price and the same deadline, sent after the cost is written down. Then report completed steps out of decision makers asked. A request that differs between segments measures the request, not the segment.
| Scenario | Likely reading | Next move |
|---|---|---|
| High frequency, little cash cost, few completed steps | A frequent inconvenience for this segment | Stop treating it as the first paying segment, or look for who does pay |
| Median cost above the margin on records, steps completed | Looks paid for this segment | Continue and widen the sample inside the segment |
| Median cost between the price and the margin, steps completed | Paid, but the price may not hold | Change the price, the offer or the sub-segment, then measure again |
| Median cost above the margin, no steps completed | The cost is real, but the offer, the timing or the buyer may be wrong | Check who owns the budget before changing the product |
Review the comparison again before choosing when any of these triggers fires:
- The mean and the median of a segment fall on different sides of the margin.
- Most counts in a segment come from memory rather than records.
- A single decision maker accounts for most of a segment's completed steps or cost.
- The observation window covered an unusual period for one segment.
- A segment holds so few locations or decision makers that one account would change the reading.
When a trigger fires, rerun the same measurement in the next window. Daniel's go-to-market mentoring page says: 'Choose what to record each week: conversations, agreed next steps, paid pilots and reasons deals stop.' That is his own description of his method. The scorecard applies the same idea to segments: record completed steps and refusals, not impressions.
A hypothetical comparison: frozen water lines at horse barns
The following is a hypothetical example with invented details. It illustrates the method and does not describe a real business, customer or measured result.
A founder plans a winter service that keeps barn water lines and field troughs from freezing, at an intended $600 a year per barn, with a required margin of twice the price, or $1,200. On the first visit, the founder watches a stable hand carry two buckets along a muddy track across a frosty paddock toward the far fence, because the line to the field trough froze overnight. The founder compares two segments, fixed in advance by who does the chores: boarding stables, where paid staff do them, and private barns, where the owner does them. One occurrence is a morning on which at least one water line or trough is frozen and water has to be carried by hand. The window is the same 4 weeks, from mid-January to mid-February, for every barn. The founder assumes a freezing season of about 12 weeks, so each count is multiplied by 3 for a year.
At the boarding stables, managers said each freeze morning adds about 1.5 staff hours, and payroll shows $20 an hour, so one freeze morning costs about $30 in paid time. At the private barns, owners said each freeze morning takes them about half an hour, which they do themselves and do not pay for.
| Barn | Who decides | Freeze mornings in 4 weeks | Yearly cash cost to the barn |
|---|---|---|---|
| Boarding A1 | Operator X | 9, seen in timesheets | $810 |
| Boarding A2 | Operator X | 12, seen in timesheets | $1,080 |
| Boarding A3 | Operator X | 10, seen in timesheets | $900 |
| Boarding A4 | Operator Y | 8, seen in timesheets | $720 in staff time plus a $1,600 burst pipe repair last winter, seen on an invoice, for $2,320 |
| Boarding A5 | Operator Z | 11, seen in timesheets | $990 |
| Private B1 | Owner | 18, seen in a chore calendar | None found; about 27 owner hours a year |
| Private B2 | Owner | 20, said | None found; about 30 owner hours a year |
| Private B3 | Owner | 15, seen in a chore calendar | None found; about 22.5 owner hours a year |
| Private B4 | Owner | 22, said | None found; about 33 owner hours a year |
| Private B5 | Owner | 19, said | None found; about 28.5 owner hours a year |
The founder then asks every decision maker for the same step: a $100 deposit to reserve a pre-winter installation slot, due in 2 weeks. The segment summary reads as follows.
| Measure | Boarding stables | Private barns |
|---|---|---|
| Locations and decision makers | 5 barns, 3 operators | 5 barns, 5 owners |
| Median freeze mornings and range | 10, from 8 to 12 | 19, from 15 to 22 |
| Counts from records | 5 of 5 barns | 2 of 5 barns |
| Median yearly cash cost per barn and range | $990, from $810 to $2,320 | None found |
| Mean yearly cash cost per barn | $1,220 | None found |
| Median owner hours a year | Not applicable; staff are paid | 28.5 |
| Deposits paid per decision maker | 2 of 3 operators: X for all three barns, and Z | 1 of 5 owners |
| The same deposits counted per barn | 4 of 5 | 1 of 5 |
Ranked by frequency, the private barns come first: their median is almost twice the boarding stables' median. Ranked by mean cost, the boarding stables clear the $1,200 margin. Both rankings mislead. The private barns show a frequent inconvenience: no cash cost was found, the time is the owners' own and one paid deposit is one account. The boarding stables show a paid problem on records, but their median of $990 sits above the $600 price and below the margin. The mean crosses the margin only because of the $1,600 repair at a single barn. Counting deposits per barn would also overstate commitment, because Operator X made one decision for three barns.
The founder keeps the assumptions next to the result: that these 4 weeks are typical of a freezing season of about 12 weeks, although winters vary; that 1.5 staff hours per freeze morning is fair, since it was said, not timed; that the staff hours would really be saved or moved to other paid work; that last winter's repair is a fair yearly figure for that barn, which is doubtful; that the owners' memory counts are close; and that 5 barns per segment say something about each segment. Five barns are no verdict on either segment. The reading: continue with boarding stables, but change something before widening the sample, either the price or a narrower sub-segment such as barns that rely on field troughs; stop treating private barns as the first paying segment; and rerun the same measurement on more boarding operators in the next cold spell.
Decide what to continue, change, or stop
Turn the scorecard into one of three decisions per segment, and date it.
- Continue when the median yearly cost per location clears the margin on figures that were seen, and decision makers completed the costly step. Widen the sample inside the segment before building for it.
- Change when the cost is real but sits between the price and the margin, when commitment comes from a single decision maker, or when the mean and the median disagree. Adjust the price, the offer or the sub-segment, and measure again the same way.
- Stop treating a segment as the first buyer when it leads on frequency but shows no cash cost and few completed steps. Keep its records, because the same problem may be paid for by a different buyer.
Severity is not segment size. The SBA guide lists 'Market size: How many people would be interested in your offering?' as its own question, separate from 'Demand: Is there a desire for your product or service?' A severe problem in a small segment and a mild problem in a large one call for different decisions, so keep size out of the severity scorecard and weigh it afterwards. To turn the chosen segment's committed accounts into a definition that sales can use, see How to Turn Customer Evidence Into a Narrow ICP for the next step.
Common mistakes and limits
- Defining segments after reading the answers. The segments then fit the data by construction.
- Comparing different windows. A segment measured in a busy season looks more severe than one measured in a quiet one.
- Counting accounts instead of locations, or locations instead of decisions. Frequency and cost belong to sites; commitment belongs to the person who decides.
- Reporting only the mean. One extreme location can carry a whole segment past the margin.
- Pricing owner time at an assumed rate. Unpaid hours are real, but they are not a budget.
- Asking segments for different steps. A cheaper or easier request in one segment makes it look more committed.
- Ranking by frequency. The segment that complains most often can be the one that pays least.
The method has limits. It suits business buyers with records and an identifiable decision maker. With a handful of locations per segment, the comparison is a judgment about what to test next, not an estimate of demand. In this guide, the Pew, SBA, Census Bureau and NIST statements are independently sourced facts, Daniel's wording is his own description, the scorecard, the rules and the readings are editorial opinion, and the horse barn case is hypothetical.
Related guides and next step
- The wider build, narrow or stop decision: Validate a Startup Idea Before Building
- Choosing between candidate problems for one segment: Is Your Startup Solving a Problem People Will Pay to Fix?
- Running the test inside one segment: How to Test a Customer Problem Before Building a Product
- The blank sheet for one account: Problem Validation Scorecard: Pain, Frequency, and Budget
- When praise outruns evidence: Why Positive Feedback Is Weak Evidence of Demand
Daniel's go-to-market page says the first session is free and takes 30 minutes. Book a startup mentoring session to review your problem validation decision with your actual evidence and constraints, and bring the segment scorecard with the records behind each figure.
Frequently asked questions
How do you compare problem severity across customer segments?
Fix each segment by an observable trait before collecting data, then measure every segment the same way: one occurrence definition, one observation window and the same unit. Count frequency and cost per location, and commitment per decision maker. Compare medians and ranges, keep evidence tags beside each figure, and rank a segment ahead only when it leads on cost and completed commitments, not on frequency alone.
What evidence should a founder collect for problem validation?
For each location, a count of occurrences in a shared window taken from a log, timesheet or calendar; the yearly cost of the current workaround with its largest item checked against an invoice or payroll; and owner hours kept apart from cash. For each decision maker, whether the same costly step was completed by the deadline. Tag each figure said, seen, done or assumed, and keep refusals and zero counts.
What are common mistakes in problem validation?
Defining segments after reading the answers, comparing segments measured in different periods, counting a multi-site owner as several decisions, reporting only the mean, pricing unpaid owner time at an assumed rate, asking segments for different steps, and ranking segments by how often they complain instead of by what the problem costs them and what they commit.
Should a founder use the mean or the median when comparing segments?
Use the median and the range, and show the mean beside them as a check. The NIST/SEMATECH e-Handbook of Statistical Methods says extreme values in the tails distort the mean but not the median, which is based on ranks. With a handful of locations per segment, one large repair can move the mean past a decision line, so find the location behind any gap and check it against its record.
Sources
- Writing Survey Questions, Pew Research Center
- Market research and competitive analysis (Plan your business guide), U.S. Small Business Administration
- Statistics of U.S. Businesses Glossary, U.S. Census Bureau
- Measures of Location, NIST/SEMATECH e-Handbook of Statistical Methods
- Go-to-Market Mentor and GTM Strategy Advisor