Not all optimization vendors are equal … and more importantly, not all vendors that claim to have decision optimization even have it (as their systems barely qualify as decision support). It’s important that you be able to distinguish between the relative strengths and weaknesses of the different products, as well as how much strength you really need, if good decision optimization is one of your driving reasons for selecting a (new) eSourcing solution. (And, by the way, as I have been stating since day one, it should be!)
The key with optimization is buying just what you need in the majority of your sourcing events. Optimization is still expensive compared to some other solutions, but more importantly buying too much power could severely impact your potential ROI (as it not only costs more but takes more manpower to use), and buying too little power will be equivalent of flushing that investment down the drain as it won’t solve the majority of your problems. I’m using the word “majority” because there is no general purpose decision optimization product for sourcing that will handle all of your events and solve all your problems. As with just about everything else in business, it’s the 8020 rule. The best solution is the one that solves as close to 80% as possible at a cost of ownership that maximizes your ROI multiple. You can always do onetime projects with bestofbreed providers or specialist outsource providers for those projects in the remaining 20% where there is enough of a savings opportunity.)
Before I get to the question list, I should point out that it’s almost impossible to cover every question, as many of the questions you should be asking depend on the answers you receive to your first few questions, but I think the question list below is a good starting point. So, without further ado, here’s the basic list!
 Does your product meet the four critera for strategic sourcing decision optimization as outlined in the Strategic Sourcing Decision Optimization wikipaper (initially authored by the doctor, the allknowing optimization king^{*1})? Specifically, does it support the following:
 Sound & Complete Solid Mathematical Foundations
such as simplex algorithms and branchandbound;
many simulation and heuristic algorithms do not guarantee analysis of every possible solution (sub)space given enough time, and, thus, are not complete in mathematical terms  True Cost Modeling
many bidders bid tiered bids, discounts, and fixed cost components – the model must be capable of supporting each of these bid types  Sophisticated Constraint Analysis
At a minimum, the model must be able to support reasonably generic and flexible constraints in each of the following four categories Capacity / Limit
allowing an award of 200K units to a supplier who can only supplier 100K units does not make for a valid model  Basic Allocation
you should be able to specify that a supplier receinves a certain amount of the business, and that business is split between two or more suppliers in feasible percentage ranges  Risk Mitigation
let’s face it – supply chains today are all about risk management, and you should be able to force multiple suppliers, geographies, lanes, etc. to mitigate those risks without specifying specific suppliers, geographies, lanes, etc. to take advantage of the full power of decision optimization  Qualitative
A good model considers quality, defect rates, waste, ontime delivery, etc.
 Capacity / Limit
 Whatif Capability
The strength of decision optimization lies in whatif analysis. Keep reading.
 Sound & Complete Solid Mathematical Foundations

Does it support the creation of multiple whatif scenarios and does it simplify the creation of these scenarios?
The true power of decision optimization does not lie in the model solution, but the ability to create different models that represent different eventualities (as this will allow you to hone in on a robust and realistic solution), to create different models off a base model plus or minus one or more constraints (as this will help you figure out how much a business rule or network design constraint costs you), and to create models under different pricing scenarios (to find out what would happen if preferred suppliers decreased prices or increased supply availability).  How fast is it for different average model sizes and can performance be tweaked?
Optimization takes what it takes. That being said, if one solution takes an average of 1 hour for an average scenario, and another solution takes 10 minutes, all things being equal, if you have compressed sourcing cycles, the 10 minute solution might be better. Emphasis on “might”. This is only true if the faster solution is of the same quality – some models, and some solvers, sacrifice quality and accuracy for speed. The best solution will let you trade off “tolerance” and accuracy for speed. Sometimes it’s easy to get within 1% or 2% in a few minutes, even though that last 1% or 2% could take hours. On a model with low total savings potential, getting within 1% may be enough. And when trying to hone in on the right whatif scenario, it’s nice to get within 1% quickly and then allow the right scenario to run to completion over night after you’ve quickly analyzed halfadozen scenarios and settled on your preferred scenario. Thus, tweaking ability is very important.  If it supports “realtime” is it “true” realtime or “near” realtime.
Thanks to significant advances in processor and hardware performance as well as offtheshelf optimizer technology (like ILog’s CPlex), it’s now possible to rapidly rebuild and resolve moderately sized models using offtheshelf modeling languages in seconds, allowing for eauction tools that keep the model relatively small and simple to incorporate decision optimization in nearrealtime by simply rebuilding and resolving the model every 3060 seconds (depending on modelsize) on a highpowered dual or quad core server with an appropriately configured and optimized CPlex 10. However, this is NOT true realtime optimization and could rapidly break down if the model gets too big or too complex. (For example, realtime optimization requires the ability to merge model construction and model solution in such a way that a new bid can be introduced as a parameter change that does not require the optimizer to rebuild the sparse model matrix and start the solution process over from scratch.) 
Describe two or three scenarios you have encountered where you could not model the situation exactly for companies in our vertical, how you worked around the issue, and how accurate the result was.
No optimization model can handle every realworld scenario 100% accurately. If a vendor representative says so, he’s either lying through his teeth or not competent enough to be selling the product. (Note that: I’ll have our support expert get back to you on that is a good answer from an average sales representative.) This is about the only way to get a decent idea of how appropriate the tool is for you. If the scenarios were complex and the constraints based on business rules you hardly ever, or never, use, then the solution is probably okay for you. If the scenarios were simple and the constraints based on business rules you use all the time, it’s probably not the tool for you. 
Can we do a pilot project before committing to a long term license?
If you like what you hear, but are still unsure, or are having problems getting the budget approved, a pilot is often the way to go! (Note that I did not use the word “free”. You should be willing to pay for services at a rate that is sufficient to cover the provider’s cost for this pilot – especially considering that many of the companies that offer affordable optimization offerings are only able to do so because they keep their costs and overheads down – and if they gave free services away to everyone who requested a free pilot, they would have to increase their costs, and that would be a detriment to everyone, including you, in the long run.) 
We’re having problems understanding how this fits into our business or what the best solution for us is. Would you be willing to demo your solution to, and answer questions from, our consultant who understands both our needs and decision optimization technology?
Let’s face it – just like the right decision optimization tool can deliver huge savings multiples on your investment (10X or more), the wrong tool will simply represent a six (or seven) figure cost that yields little return. If you can’t tell the difference, and there’s no shame in admitting you can’t if you’ve never used this type of technology before, then you should bring in a consultant who can to help you select the right technology, and ensure you are appropriately trained on it, until you are self sufficient and saving an average of 10% to 12% per project put through the tool.