13 Reasons your RFP Scoring Sucks

Today we welcome another guest post from Brian Seipel a Procurement Consultant at Source One Management Services focused on helping corporations understand their spend profile and develop actionable strategies for cost reduction and supplier relationship management. Brian has a lot of real-world project experience in supply chain distribution, and brings some unique insight on the topic.

The most thorough, best designed RFP questionnaire counts for nothing if Procurement can’t interpret the results. Proper submission scoring is critical, yet many Procurement Pros commit at least a few mistakes that seriously damage their ability to assess RFP responses.

I’ve seen my share of such mistakes over the years, and work with clients to clear them up before it’s too late. I’ve included the worst offenders, “The Unlucky 13”, below.

How many did your last RFP fall victim to?

Evaluating Questions

1. Questions aren’t weighted (or aren’t weighted properly).

Not every question is created equal. Consider how important one response is versus another. Critical questions should receive the lion’s share of total weight. I recommend starting at a high-level, assigning weight to each category of questions. Once done, delve into each category to distribute this weight to each individual question.

Not every question needs to be scored – some are for information gathering only. However, if you notice too many unscored questions, evaluate whether they all need to be included in the first place.

2. “Kill switch” responses aren’t treated as such.

On the subject of weight, some responses are so heavy that the wrong answer can (and should) disqualify a participant out of the gate. If an unacceptable answer invalidates a proposal, don’t bother weighting it – call out the answer as grounds for dismissal.

For example, one critical question may ask for confirmation that a respondent can handle required volumes. If any responses indicate a supplier can’t, no amount of weight would suffice – they simply are no longer viable.

3. Scoring is overly simplistic.

True/false questions are easy to understand and score, but too many of these cause problems in the long run. Odds are your suppliers will end up looking too similar if the bulk of responses fall into simple yes/no buckets.

4. Scoring is overly complex.

On the other side, some scoring systems end up too complex to be reasonably applied. I’ve seen scores range from 1 to 20. On paper, this appears to allow fine-tuned scoring. In reality, I’d challenge anyone to properly differentiate a score of “12” from “13.”

Evaluating Responses

5. Questions from participants go unanswered.

Your questionnaire may seem clear to your team, but chances are good that one or more participants either don’t understand your intent or don’t have the background information from you to properly answer.

Every RFP should include chances for Q&A with participants. If you don’t provide this opportunity, responses will hinge on assumptions made by participants – enough assumptions, and the end result may not align at all with your requirements.

6. Questions to participants go unasked.

The same is true on the other end. If a response is unclear to the scorer, then clarification should be sought. Otherwise, the scorer is left to make assumptions in order to interpret a response.

7. The wheat wasn’t separated from the chaff.

Anyone who’s ever scored a Marketing RFP will be familiar with this concept. Ever read a 200 word reply to a question, only realize at the end that the participant never gave a direct answer? Quantity does not equal quality – a detailed non-response is still a non-response.

Evaluating the Scoring Process

8. Clear criteria aren’t provided to scorers.

Simply providing a scoring scale isn’t enough. If you ask for a score of one to five, be sure to provide concrete direction on what constitutes a one versus a five and every point between.

9. Too few scorers are included.

The more stakeholders involved in scoring, the less likely results will be thrown by huge score discrepancies. The team in charge of scoring should encompass any stakeholders who would interact with the supplier or the product/service in addition to Procurement.

10. Score results are averaged blindly.

As a counterpoint to the above, don’t simply average all scores together at the end of the initiative. Large discrepancies in scores may indicate that two or more scorers viewed either the question or response (or both) differently. Use big discrepancies as a flag to ensure everyone is on the same page and revise accordingly.

11. External factors influence results.

Score only what it within the questionnaire. Don’t award ghost points to an incumbent based on their years of service. Likewise, don’t give an artificial boost to a hungry alternate because they came in competitively on pricing. There will be time later to consider outside elements – for now, stay focused on specific questions and responses.

12. Internal factors influence results.

“What? Dave’s team gave these guys a ‘nine’?!” “Don’t worry about it – just give them a ‘two’ to even the score out.” I wish I made this example up. I did not. I’ve worked with stakeholders who doctored their own scores to offset other scores that they disagreed with. Needless to say, this artificial tampering helps nobody.

13. Scoring lacks consistency from one response to another.

Here’s a fun way to screw with your team. Give them a pop quiz by asking them to rescore one of their first questions right after they finish scoring all responses. I’d be willing to bet on the outcome – the scores won’t match. Maybe by a little, possibly by a fair margin. When we’re evaluating half a dozen or more participants by scoring potentially hundreds of questions… it’s easy to get fatigued or change your mindset midway through.

Many people like to score one participant fully, then moving on to the next. I recommend scoring on a per-question basis instead. Take a question, and score the response from each participant down the line. Repeat for the next question. So on, so forth. This way, you’ll be in the same frame of mind and consider each response on the same standing.

Do your RFP justice – you worked hard to develop it and marshal participants through it to the end. Before working through responses, sit down with your team and review your strategy for evaluating the results. And make sure everyone is on the same page when it comes to avoiding the mistakes above.

Thanks, Brian!

One Year Ago Today …

The United States began its plunge into the new darker ages when it withdrew from UNESCO — the United Nations Educational, Scientific, and Cultural Organization.

Today, progress is global. Supply chains are global. Relationships are global. And problems are global.

What good, and in particular, what enlightenment can possibly come from withdrawing from an organization, with 193 member countries no less, whose defined purpose, as clearly outlined in Wikipedia, is to contribute to peace and security by promoting international collaboration through educational, scientific, and cultural reforms in order to increase universal respect for justice, the rule of law, and human rights along with fundamental freedom proclaimed in the United Nations Charter.

Now, I’m not defending the success, or achievements of UNESCO, or saying they are the most effective at what they do, or anything along those lines.

I’m deploring the fact that any country today could feel that, regardless of their viewpoint on the effectiveness of the organization, could not stay involved just in principal — given that with the exception of this country and one — ONE — peer, pretty much every other country is a member state.

Just look at history to see what happens whenever a country, any country, turns insular. The world leaves it behind.

And while leaving UNESCO is not the same as closing all the borders, it’s not a step forward. It’s a step backwards. And citizens should be writing their senators and complaining bitterly.

And yes, SI is aware of the publicly stated reason(s) why the US is withdrawing, and this just makes it worse. UNESCO is not meant to be a political platform. ‘Nuff said.

iZombie … Why?

So, hopefully now you understand why we’re all zombies. Procurement was supposed to be dead and buried two years ago (search the archives), but instead of the Procurement Phoenix rising from the ashes and blazing a new course, nothing, and we mean nothing, has changed.

Day by day we’re going through the same old motions, using the same old processes, dealing with the same old issues, on the same old platforms.


First of all, why are we using the same old processes. While the intent of Procurement has not changed since the first Purchasing Manual was written back in 1887, the nature has. We’ve gone from vertically integrated companies with relatively local supply chains (where only raw materials were imported and only when necessary) to horizontally distributed supply chains where everything can be imported and exported through every step of the chain and where there can be three or four levels in the chain.

Secondly, why are our suppliers still okay with this? Presumably we’ve learned a few things in the past one hundred and thirty years or so? Presumably we have better processes for managing the entire source to contract to delivery to return to rinse and repeat cycle. And presumably we can use that knowledge in our dealings with them and make things better for them as well as us.

Thirdly, and most importantly, why aren’t we using platforms that enable these better processes both for us and our suppliers?

Well, unfortunately, we already know part of the answer here. As per our recent series on how 2020 is Fast Approaching (and that you better get on your tech capabilities), modern platforms are nowhere near where they were supposed to be. That does limit what you can do and the experience you can provide them.

But that’s not the whole answer. There’s also the fact that most of you aren’t on the best platforms you could be on, most of which are very limited in supplier relationship capabilities — which is key to building supplier relationships and making them want to use the platforms.

But you can’t put it all the vendors, as there’s also a third side to this story. You’re not insisting enough that your vendors get better. And you’re definitely not doing it from day one. Threatening to switch at renewal time isn’t much of a threat to the vendor when they know the sunk cost and the cost of change is high and that you’re not likely to be allowed to do it.

If you want to scare them, you have to take it up a notch. Threaten to tell the supplier the real reason their experience is so poor and all the faults of the platform you’re using — remember, your suppliers need platforms too. But that’s not the real answer (especially if you didn’t read the contract carefully and/or you agreed to a no disparagement clause).

The real answer is to make sure, at contract (renewal) time, you put a mandatory platform improvement clause in the contract with necessary features and functionality the supplier must deliver, on a schedule, or you get to either a) use the platform free until they deliver or b) leave at any time with no penalty and take ALL of your data with you, which they must output in its entirety in a well defined and documented database and/or (x)MXL schema that you can take anywhere.

Then you’ll get the platform you need to support more modern practices which may allow you to eliminate the same old — tactical based — issues, which result from poor platforms and poor processes.

Will it solve all your problems? Heck no. But maybe, just maybe, you’ll have a lot more time to focus on the real fires … instead of constantly having to deal with ventilating the smoke.

How? To be continued …

iZombie Interlude

In popular culture, a zombie is defined to be a person who has died and been reanimated in a manner that typically causes them to lose intelligence and, sometimes, even, sense of self. Sometimes its because a voodoo doctor or other evil personage re-animated them. Other times its because a virus, often created by a mad scientist, had a very adverse effect and created an infection that spread (like wildfire) and create half-living / half-dead creatures that are, for all intents and purposes, zombies.

But that’s not the only definition of a zombie. Another definition of zombie is a person held to resemble the so-called walking dead, included, but not limited to, an automaton. An automaton that is indistinguishable from a person that repeats the same well-defined tactical tasks over and over again, day after day after day.

A person that, with the right sustenance (brains in popular culture, or coffee and painkillers in others), can actually maintain semblance of self and intelligence, even if such semblance of self and intelligence is rarely seen during the day to day performance of their tasks.

A person that, in all likelihood, works in an organizational back-office job in an enterprise using outdated technology and inefficient processes, like your average Procurement organization.

So, when SI is saying that iZombie is a good name for our profession that was supposed to be dead and buried two years ago, but is still going … on auto-pilot. Automatons we are.



iZombie: A Prelude Part II

12:59 pm – You settle into your seat in the meeting room, in preparation for an hour of arguing, finger pointing, and general non-performance between the marketing rep, engineering rep, finance rep, and the customer success rep (that represents the account management team).

First off, the customer success rep is p!ss3d that the incumbent supplier, that is supplying four of their five top customers is being invited back as all of the customers are complaining about product quality. The supplier did send out a few bad shipments, but has since re-instituted the mandated quality assurance processes and allowed you to place a quality rep at their site that reports to your organization. Quality has increased to minimally acceptable levels on the last two shipments and things are on the up and up. (But since each customer only gets a shipment every three months, those four customers didn’t get their yet, scheduled for the next two batches.) And the customer success rep would know if engineering had a way to update them.

Next up, the finance rep wants to know why the two lowest bidders have been disqualified because the organization needs to lower costs by 10% and those are the only two suppliers likely guaranteed to do so on the project. It’s pretty obvious why based on the product ratings from engineering and customers and the historical quality ratings across projects, but apparently finance couldn’t be bothered to figure out how to run the reports.

Then the engineering rep wants to know why they can’t just keep using the incumbent, who has already agreed to a 5% price reduction to make up for their quality failure. Especially since Engineering put all this time into qualifying them and building the relationship. (Unaware that this supplier has also performed poorly across sister companies, which you are aware because the PE firm that bankrolls you collects cross-organizational data and builds it into the risk reporting feeds you get monthly — outside the platform.)

Finally, marketing is up in arms that you are not looking for suppliers that can deliver products with new and exciting features they can sell.

Thirty minutes of complaining, ranting, and basic Q&A pass before you can even get down to business.

The RFP was scheduled to go out today, on final review, but the supplier list has yet to be approved, hence the reason for this after-the-last-minute meeting. The incumbent supplier, three previous bidders, and six potential new bidders were invited to the RFI, two new bidders didn’t respond, and two didn’t meet the quality cut. You were ready to send it out to six, but customer success still feels their grievances haven’t been heard on the incumbent, engineering says they will disqualify the previously invited bidders for the same reason, and finance feels the two new suppliers, based on their bids in other projects, won’t be competitive.

Finance wants more suppliers, you need suppliers engineering will consider, and without at least one supplier that tickles marketing, you’ll hear nothing but moaning for months.

So you need to do another discovery project. There’s another 20 hours and 2 more weeks down the drain. And an addendum to the report you just gave your boss less than two hours ago – damn!

Anyway, that’s tomorrow’s project. The meeting has ended, and you have 30 supplier e-mails that you need to review today, or you know they will be repeated tomorrow — plus, one ranting and raving phone call a day is enough.

2:10 pm – Into your tenth supplier e-mail, most inquiring about invoices, event status, etc. — and other inane questions that could be answered through their supplier portal, as awkward as it may be, if they’d just learn to use it, you hit an e-mail from your key widget supplier informing you that the shipment that was supposed to go out yesterday is not complete as they have been waiting on a steel shipment for two weeks. Yikes! If you don’t get widgets in 21 days, your production line grinds to a halt! That will cost the company millions.

3:45 pm – After an hour and a half of frantic calls, you find another — off-contract — supplier that can supply a substitute shipment of widgets that, while not preferred, will work with a few minor production line tweaks and keep you going for six more weeks. If you can’t get regular shipments out of your supplier within that timeframe, you’ll be in trouble. But that’s tomorrow’s problem. For today, you still have 20 supplier e-mail and almost two dozen stakeholder e-mails to get through.

4:25 pm – The remaining supplier e-mails are not critical, a few require clearing up with AP and engineering, but that can be taken care of later in the week.

4:35 pm – Most of the stakeholder e-mails are answers to your inquiries, inquiries as to why Sourcing takes so damn long, or indicates of delay. The usual affair. Time to do a quick check on existing project status.

4:50 pm – Three projects need addressing in addition to the cog project. The axle project has questions from suppliers that need to be answered so they’ll bid. The cylinder project still needs RFI scoring from stakeholders so it can go to auction. The support team project needs more bids for key contingent workforce roles and you don’t understand why your chosen providers aren’t bidding. Fire-fighting for the next few hours.

7:15 pm – You finally get all the questions answered in the axle projects — and hopefully the suppliers will bid in the next few days; you send reminders to all the stakeholders and leave the ones who don’t read e-mails voice mails; and you call all of the contingent workforce providers in the earlier timezones where they are still in the office and discuss how important it is that they bid on the current project if they want to continue to bill you, and most agree to provide more bids tomorrow.

No real progress on anything, but you made it through today.

7:20 pm – You exit the building.

7:21 pm – You realize the headache you obtained during the stakeholder meeting this afternoon is now, as usual, unbearable so you pop a few extra strength Aleve so you can make make it home, have dinner, and rest up for another day … which likely won’t be that much different from today.