Re: the posting by Mr. J. H. Steiger of UBC in response to my
earlier posting.
>Re: the posting by Mr. M. Palij of NYU.
>
> Mr. Palij writes:
>
>>In the course of reviewing the PC magazine article that
>>Ms. Czyz referred to in her earlier posting to this list
>>I came across a couple of inaccuracies that I thought
>>should be made known.
>
> Mr. Palij says:
>
>>Ms Czyz misrepresents the ratings given in the PC Magazine
>>article. First, each package was rated on three dimensions
>>(Czyz statement implies there was one rating).
Mr. Stegier writes:
> Systat certainly seemed to believe it came out on top. It has referred
> to this Editor's Choice rating several times in its advertising
> and promotional materials.
The issue here is not what Systat has said but what Ms. Czyz
has said. The PC Magazine article is available for all to
read and the facts are independently verifiable.
> I did not keep a copy of the PC Magazine review (for reasons that will
> become apparent later in this posting). It is my distinct recollection that
> 1. There was an overall rating (Editor's Choice?).
You are in error if you think of overall rating as a single score,
such as the dosmark or winmark scores used to rate PC hardware
performance. There were three dimensions, represented by a
four point scale (poor, fair, good, excellent): data management,
basic statistics, and advanced statistics. Scores on these served
as the basis for the Editor's Choice decision.
> 2. Systat and SPSS were co-winners.
SPSS for Windows received the highest scores on the three dimensions
and Systat tied with another SPSS product as shown in table below
(Statistica included for comparison):
SPSS/WIN v5.0 SPSS/DOS v5.0 Systat Statistica
Data Man. 4 (out of 4) 3 3 2
Basic Stat 4 3 4 3
Adv. Stat 4 4 3 3
Systat had one set of ratings for both Dos and Windows and
Statistica was rated only for the Dos version (the numbers come
from pp 259, 284, 285 of the 11 May 1993 PC Magazine).
> So it appears Mr. Palij is engaged in a hair-splitting exercise here.
> I cannot see how anyone might be misled by her comment.
Reread her comment, which I reproduce below:
:[Ms. Czyz wrote]
:>THE FOLLOWING FACT RELATED TO THIS REVIEW WILL
:>WELL ILLUSTRATE HOW STATSOFT HAS AVOIDED EVEN TO GIVE
:>ANY IMPRESSION OF ATTEMPTING TO INFLUENCE IN ANY WAY
:>THE PROCESS OF REVIEWING ITS SOFTWARE. Until now
:>StatSoft has never made public (or even reported to
:>the editors of PC MAGAZINE), the fact that Mr. Grant
:>Blank--the reviewer (whose lukewarm evaluation of
:>STATISTICA allowed SYSTAT to get the highest rating in
:>that comparative review) was a close associate of
:>SYSTAT.
She subsequently refers to Systat as getting the highest
rating again and again. I will leave it to the reader to see
how one might be misled by this comment.
> While we're on the subject of accuracy, consider the following.
> I have been led to believe, by comments in PC Magazine, that
> Editor's Choice is a dichotomous award. You receive it or you do
> not.
>
> Yet Mr. Palij wrote:
>
>>SPSS/WIN had the highest ratings
>>and was presented first in the Editor's Choice column.
>
> This appears to imply a rank ordering on both the "ratings" and the
> Editor's Choice. Yet I assume SPSS was "presented first"
> because it comes before Systat in alphabetical orderings.
Again, you are wrong, as the table above clearly indicates.
Both were selected as Editor's Choices but SPSS/WIN outscored
Systat on 2 of the dimensions.
> Mr. Palij's statement might be considered misleading.
> Perhaps he could clarify.
Clearly Mr. Steiger's problem is that does not have a copy of
the review article handy. Hopefully I have presented it
clearly enough so that even he will now understand it.
> Mr. Palij has presented us with some interesting additional information. I
> do not recall this information appearing in the original PC Mag
> article, so it would be interesting to know his source. If true,
> it certainly provides some food for thought.
Prehaps Mr. Steiger should go back to original sources rather
than relying upon his memory. It would save some bandwidth.
As I have said before, the article is available to all for
independent verification. All of the information I have comes
from the article.
> Mr. Palij wrote:
>
>>Systat was reviewed by Sheryl Canter and Charles
>>Kadushin who, as far as I know, have no relationship to
>>Mr. Blank. .......Canter and Kadushin also
>>did the reviewing of the three SPSS packages reviewed.
>>SPSS/WIN was also presented before Systat in the Editor's
>>Choice column.
>
> If Mr. Palij is correct, PC Magazine had one pair of reviewers review
> SPSS and Systat, and an entirely different reviewer review Statistica.
It is surprising that this should come as some sort of shock
to Mr. Steiger. The reviewers are identified on page 228 and
the reviewer associated with each individual review is
identified after the heading of the package. Ms. Canter's and
Mr. Kadushin's names are given on page 259 (for the SPSS
review) and on page page 284 (for the Systat review). This is
public information. I wonder why these concerns did not
emerge at the time he read the article.
> The table below shows the experimental design and outcome.
>
> ------------------------------------------------------------------
> Package SPSS Systat Statistica
> ------------------------------------------------------------------
> Reviewer(s) Canter Canter Blank
> Kadushin Kadushin
> ------------------------------------------------------------------
>
> Rating High High Not so high
> ------------------------------------------------------------------
>
> Perhaps Mr. Palij could clarify for us if he really feels this is an
> appropriate way to design an objective review. I suggest we present
> the table to a class of third year Psychology undergraduates in a
> research methods class, to see if they find any flaws.
We could discuss this point but that would divert us from the
main point, namely Ms. Czyz's claim that Systat was given the
highest rating when it was not. Perhaps Mr. Steiger should
point out to Ms. Czyz why she shouldn't use the PC Magazine
reviews or, better yet, he should inform PC Magazine of the
problem if it is of such significant concern to him.
> Yet Palij seems to feel there is no reason for Czyz to complain.
Ms. Czyz was not complaining about the methodology used, she
was complaining about a conflict of interest on the part of
one of the reviewers. That should be obvious to even you at
this point.
>>Wouldn't one have to posit a conspiracy
>>theory so that (a) Statistica would be rated poorly and
>>(b) Systat be rated highly?
>
> No conspiracy theory is necessary. Simple knowledge of the
> principles of experimental design might suffice. Things like "rater
> effects" come to mind immediately..... One does not have to posit
> bad faith, or a bad review, on Grant Blank's part.
>
> The simple point (apparently missed by Mr. Palij) is that the review
> was not done correctly (according to elementary principles of
> experimental design), AND Mr. Blank had assisted in the design of
> part of Systat. Since Systat was involved in a public quarrel with
> Statsoft, that MIGHT have compromised his objectivity toward Statistica.
> Yet StatSoft had not contested the result.
These statements, of course, have no real bearing in the
original issue, namely, that Ms. Czyz claimed that Systat had
gotten the highest rating. Since Mr. Blank has made his
appearance I will leave the accusations regarding him to be
answered by him.
> These facts were posted by Maria Czyz in response to comments by
> Alan Best (which I will reprint for below, in case the reader
> is unaware of their tone and content).
>
> Best had written:
[Steiger's heavily editted comments of Al Best deleted.
These simply have no relevance to the question at hand,
that Ms. Czyz made incorrect statements about Systat's
rating in PC Magazine. I have complete files of the
discussion so far and will email copies of the original
intact message to anyone who would like them.]
> Maria Czyz's posting had two purposes
>
> (1) It challenged Best to support his allegations with facts,
>
> (2) It presented a case where of a situation
> where StatSoft might well have complained about an
> unfair review, but did not.
The purpose of my posting was to identify the error that Ms.
Czyz made in referring to Systat's rating in the article.
Mr. Steiger clearly does not want to deal with this issue.
> Rather than damaging Czyz's case, Mr. Palij adds further support.
> Specifically, if PC magazine had randomly assigned different reviewers
> to different packages, there might be some faint claim that the
> review is useful. Thanks to Mr. Palij, we find that, in fact, the
> design of the review was unbalanced in an extremely interesting way.
It seems that Mr. Steiger simply cannot keep his mind on the
relevant issue, namely, Ms. Czyz claimed that Systat had
gotten the highest score in the PC magazine review. This
was the point of my previous posting. Perhaps Mr. Steiger is
responding to some one else's posting.
> Mr. Palij concluded:
>
>>The above comments are misleading and should be retracted
>>by Statsoft. Specifically, SPSS/WIN had the highest ratings
>>and was presented first in the Editor's Choice column.
>
> As I pointed out above, the ironic fact is that this comment is
> misleading. It implies a hierarchy in a dichotomous category.
Did Ms. Czyz claim that Systat had gotten the highest rating?
Is this statement inconsistent with the facts?
> I conclude something quite different.
>
> I believe that, unless Mr. Best can substantiate his "grapevine,"
> it is he who should be doing the retracting. Moreover, I would urge
> all participants to refrain from posting defamatory statements about
> individuals or organizations on this forum. If Mr. Best wants to write
> to all his friends and tell them his opinions, he may do so. This
> forum should be reserved for interesting technical discussion.
>
> Mr. Palij's posting has proven useful. I appreciate his providing us
> with additional information about the PC Magazine review.
Thank you for the kinds words, which I appreciate as a change
of pace from you.
-Mike Palij/Psychology Dept/New York University