One thing I believe often missing when considering vendors proposals, is how they address or show how they want to be involved in elements of the exploitation phase, e.g. inclusion of warranty period and provision of help desk support for user support and an ongoing maintenance and review programs which is typically after the project goes live.
Even though many companies may elect not to use this, it shows the vendor is confidence and committed to ensuring the project is successful and gives customers great comfort on risk from attrition and change.
In Information Management Newsletters I subscribe to , on November 9, 2010 Kumar Ramamurthy posted these 20 Questions to Ask When Choosing a DW/BI Vendor Partner For the record I have posted it here and refer you to the original. It as an excellent article.
I have summarized here what he says to look for in vendor proposals
- How well a vendor has understood the business problem and the value they bring.
- Requirements are separated into source system , data integration and BI layer
- Requirements provided have been well-dissected
- Shows depth of the vendor experience in executing turnkey DW/BI projects
- The team is led by a DW/BI-certified project manager/lead
- Black box estimates, show dependencies to realistically limit surprises.
- Vendor plan shows measures of productivity gains and efficiencies
- Vendor shows component parts of the bid, and who owns the IP rights
- Quality and performance metrics capturing included and inferred during project lifecycle
- .Rollback mechanism or running a parallel system been considered as part of estimates
- Has well-balanced test automation and automated test data creation process
- Has knowledge transfer plan to customer’s for DW/BI team
- “Code freeze” time is mandated in proposal to push changes to production systems
- Power users have guidelines to develop reports or query the data and functionality
- Vendor has mechanisms to point out the code that contributes to SLA lapses
- Project estimate in project phases to allow de-scope nicely, or adding functionality
- Project dissected for the chance to bring some phases of the project in-house,
- Risk/reward clause Inclusions indicates vendor is very confident of executing it well.
- Relevant proofs of concept cited that have been showcased and shown as well tested
- Show what elements in the proposed solution are unique differentiators
Included in his post he makes a very good point that the handoff includes a well-documented solution cookbook, run book and a meeting to pass the details on to people who will continue to support the system and functionality developed.
Here is the full post
20 Questions to Ask When Choosing a DW/BI Vendor Partner
Information Management Newsletters, November 9, 2010
Businesses are seeing a greater need today to enable fact-driven decision-making, forecasting, planning and inventory control. Request for proposals from customers seeking data warehousing, business intelligence and analytics solutions are drawing scores of responses from established, niche and boutique vendors of all sizes.
Each of these firms brings economies of scale, geographic location and partnerships with technology vendors as key differentiators within their proposal in an effort to win business from the customer. While these aspects certainly carry their own little merit to aid in selecting the software vendor, customers need a stronger criteria set that is relevant in order to evaluate these vendor proposals from common ground.
Irrespective of the size, complexity or scope of the DW/BI project, answers to the questions below serve as a good litmus test to narrow down a list of potential vendors.
1. Does the bid consider the end to end impact on the DW/BI platform in the proposed project approach? The answer to this will clearly show the depth the vendor has in executing turnkey DW/BI projects. It would also give the customer a clear indication of how well the vendor has understood the business problem and the value-add they can bring to the table.
2. Have the requirements been separated into source system changes, data integration changes and BI layer changes? A satisfactory answer to this question usually indicates that the vendor is rightfully dealing with the requirements provided, since the requirements have been well-dissected before delivering the proposal.
3. Were delivery dates independently planned or dependencies considered for each of them? If the effort for streams of work was estimated as a black box, but when the project was planned, dependencies were considered, the customer is in safe hands and will realistically see fewer surprises in the plan.
4. Has a system cutover plan been included with the proposal? Are the changes going to be done in an online or offline mode? Has downtime been accounted for in the deployment plan? Usually, inclusion of such activities in the plan and the proposal indicate that the customer is not a pure development shop alone, but will ensure that the project is in production before they deem the project complete. It shows that the vendor is willing to stand by what he or she develops and is invested in getting it right.
5. Has a rollback mechanism or running a parallel system been considered as part of the effort estimates? Not all projects go right at the first installation; failure-proof planning, if included in the proposal, shows the vendor is not afraid to face reality.
6. Has a “code freeze” time been mandated in the proposal to effectively push changes to production systems? How does the solution account for catching up with other development initiatives “in flight”? Quiz the vendor about this problem and how well “continuous integration” and “automated nightly builds” will help address this problem. Look to see if this is a part of their proposal.
7. Is there a mechanism to streamline and govern the use of new data available in the data warehouse? Has metadata been published on reporting/ad hoc query tools effectively, to be used by power users who need to use the data? Do the power users have access to a documented guideline to develop reports or query the newly published data and functionality?
8. Will the business glossary terms, their relation to columns in the database and report attributes including derivations, consolidations and aggregations be documented? Ensure the handoff includes a well-documented solution cookbook, run book and a meeting to pass the details on to people who will continue to support the system and functionality developed.
9. Is there a mechanism to point out which line of code, in the newly introduced functionality, and attributes to degradation in performance, if any? Ask for a functionality-code-performance co-relation report to ensure the vendor has mechanisms to point out the code that contributes to service level agreement lapses.
10. Does the proposed staffing model consist of BI personnel who understand data and data integration personnel who understand BI? Is the team led by a DW/BI-certified project manager/lead?
11. Can code quality be measured? Can some of the metrics (such as right first time; defect ratio per change or the quality index; performance metrics of the system; and coding metrics like review comments per change, code quality, documents for a change, etc.) be inferred from the metrics the vendor is capturing during the project lifecycle?
12. How does the vendor plan to measure productivity gains and efficiencies as his or her team gets familiar with your DW/BI environment? Are there any reusable tools, solutions or components that are considered to be built as a part of the bid, and who owns the IP rights for them?
13. How much of the testing can be automated? Typically, regression testing cycles on DW/BI projects cost more than the development effort. Having test automation and an automated test data creation process will ensure that effort is well-balanced, while reducing the total cost of the project and creating a repeatable automated test scenario.
14. How is knowledge transfer to a customer’s DW/BI team planned? This ensures that the vendor has thought about the whole project life-cycle, including transition of knowledge to the customer.
15. How much of production support and warranty is included in the proposal? This shows the vendor’s confidence in the proposed solution and how well they are prepared to support something they developed.
16. Is there an estimate view by functionality, project phases and by technology components? This gives the customer a chance to de-scope nicely, which allows for the inclusion of additional functionality and provides an opportunity to keep project costs low. This also gives the customer the chance to bring some phases of the project in-house, based on resources available. It helps the customer to cut costs, engage in-house resources effectively and keep a constant vigil on the vendor deliverables, thereby avoiding any big surprises.
17. Is there a relevant case study in the proposal that indicates the experience of having dealt with such projects before? Can references pertaining to the showcased case study be provided?
18. Are there any risk/reward clauses in the proposal? Inclusion of such a clause, usually when tied with a fixed bid made for the project, indicates that the vendor is very confident of executing it well.
19. Are any relevant proofs of concept that have been showcased on cutting-edge technology being offered in the proposal? Often, such proofs show that the vendor has a lab-type environment and an R&D division where the cutting-edge technology proposed has stood the trial of strength.
20. Lastly and most importantly, if the vendor takes the time to present what differentiates them from other vendors or what elements in the proposed solution are unique differentiators, the customer should have enough ammunition to rank vendors and award work to those who come out on top.
Ideally, this list of 20 questions will act as an acid test to rank top vendors who respond to a customer’s DW/BI RFP. A next step could be to pick the top two or three vendors, award them pilot projects in different streams and measure their system development lifecycle metrics at length. This will aid in the decision process of selecting a strategic DW/BI vendor partner.
Kumar is the Chief Technologist for the DWBI practice at Virtusa. He has over fifteen years of experience in enterprise data architecture, database related technologies, software platforms and architecture assessments. He can be reached at firstname.lastname@example.org.