This isn't really a how to - it needs rewriting to provide a solid series of steps - some of the content is better for index pages.
Evaluating Open Source
Evaluating open source software is no different from evaluating closed source software, that is, the main question is "does it meet my requirements?" However, it is also necessary to evaluate the development model and licence of the software to ensure that the full benefits of open source will be realisable. For example, software under an open source licence that does not have a clearly defined process for accepting third party contributions may be difficult to engage with should modifications be required.
Formal Evaluation Techniques
There some formal evaluation techniques that you may choose to adopt, for example:
The Business Readiness Rating evaluation model involves four steps: a quick assessment to draw up a short list of software packages to evaluate; the ranking and weighting of the selection criteria; data gathering for each criteria; calculation and publication of results. The Open Source Maturity Model is less detailed and restrictive, but as a produces quicker results, of course, cutting corners can result in an incomplete evaluation.
Informal Evaluation Techniques
Many people evaluate through informal processes, whilst this is not to be recommended in a significant procurement exercise it can sometimes be appropriate. OSS Watch maintain a number of documents that are designed to inform readers about what they should be looking for when evaluating software:
Open source for absolute beginners - an index page to our resources to introductory materials about open source and open development
Top Tips For Selecting Open Source Software - generic tips for evaluating open source software
Finding your way around SourceForge - contains pointers to specific evaluation tools within Sourceforge
Documentation issues in open source - describes the problems of creating good documentation in open source projects and thus helps point out flaws to look for