[Xen-cim] Cimtester vs. Sblim-testsuite

["Szymanski, Lukasz K" <Lukasz.Szymanski@xxxxxxxxxx>]

During last Thursday's call Jim and I talked about choosing the test suite which will be least burdensome and most flexible.  I spent some time evaluating sblim-test in the last few days in order to give both cimtester and sblim-test a fair shake. It's a somewhat close race.

I am attaching an excel spreadsheet with the details of my findings but the short story is this:

Cimtester has one input file, sblim test has one input class per test. 
Advantage: cimtester

Cimtester supports exec query from within the test.  Sblim-test does not.
Advantage: cimtester

Cimtester supports runnig external tests from within the test.  Sblim-test does not.
Advantage: cimtester

Cimtester has one output file which is not always very informative.  Sblim-test has one, nicely formatted output file per class

Advantage: sblim-test

Cimtester only catches exceptions, sblim-test will catch and interpret wbemcli errors.
Advantage: sblim-test

Sblim test has an additional set of tests to shake out consistency.
Advantage: sblim-test (see spreadsheets for more details)

I have a rudimentary script which writes the cimtester input xml file.  Sblim-test would require some more effort of that nature, but if the IBM folks are using sblim-test and have some scripts already, life might be a lot easier.  Cimtester requires some interpretation on the output end, because the errors are not as informative or well formatted.  Sblim-test has a whole suite to test consistency but it would require more work and I am not sure if the value is worth the effort. Two big strikes against sblim test are lack of support for exec query and external tests.  I think that tips the scales in favor of cimtester, but I am open to discussion.  Please let me know what you think.

Luke

<<cim_sblim_diff.xls>>

cim_sblim_diff.xls
Description: cim_sblim_diff.xls

_______________________________________________
Xen-cim mailing list
Xen-cim@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-cim