Point 1
-------
StatSoft> "... the article ... did not mention any of the unique
features of STATISTICA ... (e.g. the level of integration of numerical
results with graphics)"
PC Magazine Review sentence #1> "The integration of graphics and
statistics is one of the package's strengths." p. 283.
PC Magazine review sentence #2> "Its advanced capabilities, such as
graphics and the integration of graphics and statistics..." p. 284.
PC Magazine review sentence #3> "Its strengths are its excellent
graphics and close integration of graphics and statistics." p. 284.
Grant Blank> Three times.
Point 2
-------
StatSoft> "... the review also contained factual errors."
Grant Blank> PC Magazine does extensive fact checking. PC Magazine
staff talked with StatSoft technical support repeatedly about the facts
stated in the review. The staff made various changes as a result of
these conversations. In addition, the PC Magazine editors and I spent
hours in phone conversations with Paul Lewicki, President of StatSoft,
specifically checking the contents of the review. We changed several
parts of the review so that it reflected his understanding of his
product. We gave him essentially unlimited time to point out errors.
In the published text, there are things that he would have not said.
No one will be surprised that he doesn't like the criticisms; he is the
vendor. Criticism is different from factual errors. If factual errors
remain, neither he nor anyone else on his staff pointed them out at the
time.
Vendors of software under review typically make two kinds of errors.
(1) They claim their package can do things that it can't. This is
widely understood and usually attributed to the overstatements of a
party with financial interests at stake. (2) The second error is less
widely recognized: they do not acknowledge features that their package
actually has. The cause here seems to be that modern full-featured
software is complex and no one, not even the vendor, knows every
detail. Both these errors have occurred in every review that I've ever
participated in. Because both are common, PC Magazine does extensive,
independent fact checking. Despite the efforts of PC Magazine and
vendors, errors (particularly type (2)) still appear in reviews. In
the Statistica review, more than in other reviews, we spent hours
checking with StatSoft. Paul heard every fact in the review and, if
factual errors remain, he approved them personally.
Point 3
-------
StatSoft> "It was also disappointing that the reviewer of
STATISTICA did not include more information about the Windows version
of the program which was already available at that time..."
Grant Blank> This wording is ambiguous. The word "available" leaves
the impression that the Windows version had been released. The review
went to press in early February 1993. When I talked with Paul, I asked
repeatedly if the Windows version had been released. He repeatedly told
me it had not.
PC Magazine does not, as a matter of policy, review products before
their release. Paul was personally told of this policy by the editors.
>From the PC Magazine point of view, prerelease software is not
"available" for review. Paul sent me prerelease software and a
photocopy of a 104-page draft of a quick-reference manual. Paul said no
other manual was available. The paragraph where the review discusses
Statistica for Windows is based on this material. If Paul had more to
send me he could have sent it at any time. He sent nothing else.
Point 4
-------
StatSoft> "... allowed SYSTAT to get the highest rating ..."
Grant Blank> The packages were rated on 3 criteria: Data management,
Basic statistics, Advanced statistics. SPSS 5.0 for Windows received an
Excellent rating in all criteria (p. 259) and thus received the highest
rating in the review. Systat was rated Excellent in only 1 criterion
(p. 285). Reflecting this, SPSS for Windows is discussed first in the
Editors' Choice article (p. 229). Systat was given an Editors' Choice
award, but readers of PC Magazine know that Editors' Choice is not the
same as the ratings. One difference: ratings are given by reviewers,
the Editors' Choice is awarded by the editors.
Point 5
-------
StatSoft> "... StatSoft has avoided even to give [sic] any impression
of attempting to influence in any way the process of reviewing its
software."
Grant Blank> Paul Lewicki spent hours on the phone with me and and more
hours with the editors attempting to influence the review. Indeed, I
spent a large part of the final week of the review doing little else
but being influenced by StatSoft in the form of phone calls from its
President. In his calls to the editors of PC Magazine, Paul did his
utmost to influence the review, making detailed arguments based on his
knowledge of the wording of the review. At one point, he tried to
convince the editors not to publish the review because it would not be
strongly positive. In effect, the editor served as the referee in a
point-by-point debate between myself and Paul. Few changes resulted,
but the published text contains only language that the editor was
convinced was supported by his own exhaustive examination of the
product.
Other vendors may be surprised to read this. No other vendor had this
access.
Point 6
-------
This brings me to the final point in the StatSoft message. It alleges
that I did not report to the editors of PC Magazine that I was a "close
associate of SYSTAT" and I, therefore, engaged in unethical behavior.
I am an independent consultant and I consult on statistical and
database problems, among others. SYSTAT is one of several statistical
software companies numbered among my clients. Before the review began
other statisticians told the PC Magazine editors about my clients and,
before I was assigned a package to review, I told the editors
personally. Fully informed, the editors then decided I was the
appropriate reviewer for Statistica and made the assignment.
Had StatSoft checked with someone involved in the review they would
have been told the truth and would have avoided the embarrassment of
making a false accusation. If there is any ethical question here it
is why StatSoft would make a public accusation without checking and
without evidence.
Point 7
-------
The StatSoft message is by a vendor who wants a more favorable review.
But no good review is ever written to make a vendor happy. The focal
audience of a review is users and potential users. The goal of a
review is to help users make informed decisions. My review of
Statistica/DOS discusses the strengths of the package, like the
integration of graphics and statistics, so that people who could
benefit from it will know to look at it. It also discusses weaknesses,
like inadequate documentation, and poor handling of categorical and
character data, so that people for whom it is unsuitable will know to
stay away.
StatSoft wants to believe that I pointed out various weaknesses
because I was biased. This is false. My conclusions are based on
intensive interaction with Statistica/DOS and it's vendor, supported by
extensive evidence gathered during those interactions. The real truth
is simple, and not what Paul Lewicki or StatSoft want to hear. I
called it like I saw it; Statistica is not as good as the vendor thinks
it is.
Vendors can believe whatever they want; in advertisements they can say
whatever they want, but a reviewer's responsibility is to users and
potential users. In my opinion, the users who will be happiest with
Statistica/DOS version 3.1 are those who need to do statistics and
graphics on small datasets requiring only simple transformations and no
complex data management, and it is not suitable for analysis of
marketing or survey data. That's what I said in the review. The
review does what any good review tries to do; it praises the product
where it is strong and criticizes it where it is weak. PC Magazine
reviewed 13 statistical packages. Compared to Statistica, some are
worse but some are better.
I wrote a lukewarm review of Statistica because that's what it deserved.
Grant Blank
grant@sam.uchicago.edu