{"id":1325,"date":"2013-05-29T19:46:42","date_gmt":"2013-05-29T18:46:42","guid":{"rendered":"http:\/\/www.jutze.com\/?p=1325"},"modified":"2013-05-30T16:42:52","modified_gmt":"2013-05-30T15:42:52","slug":"null-hypothesis-significance-testing-the-fault-in-our-stars","status":"publish","type":"post","link":"https:\/\/www.jutze.com\/?p=1325","title":{"rendered":"Null Hypothesis Significance Testing: The Fault in Our Stars"},"content":{"rendered":"<p><a title=\"Why Has The Fault in Our Stars Been So Successful? - John Green's tumblr\" href=\"http:\/\/fishingboatproceeds.tumblr.com\/post\/51645711954\/why-has-the-fault-in-our-stars-been-so-successful\" target=\"_blank\">fishingboatproceeds<\/a><\/p>\n<blockquote><p>[&#8230;] The same is true on <a title=\"Amazon.com - The Fault in Our Stars\" href=\"http:\/\/www.amazon.com\/dp\/0525478817\/\" target=\"_blank\">amazon<\/a>, where the book\u2019s average rating has actually gone up a bit in the past six months (although not in a statistically significant way). [&#8230;]<\/p><\/blockquote>\n<p>Actually, the ratings have decreased in a statistically significant way (alpha &lt; .05). I used the two most recently archived pages from <a href=\"http:\/\/www.archive.org\" target=\"_blank\">archive.org<\/a>, which do not cover exactly 6 months. Still, ratings before <a href=\"http:\/\/wayback.archive.org\/web\/20130203060552\/http:\/\/www.amazon.com\/The-Fault-Stars-John-Green\/dp\/0525478817\" target=\"_blank\">2013-02-03<\/a> were higher than those after that date.<\/p>\n<ul>\n<li>Before (2110 ratings): mean = 4.76 (<abbr title=\"standard deviation\">SD<\/abbr> = 0.014)<\/li>\n<li>After (1232 ratings): mean = 4.67 (<abbr title=\"standard deviation\">SD<\/abbr> = 0.021)<\/li>\n<\/ul>\n<p>A <em>t<\/em>-test (two-sided, unequal variances) yields <em>p<\/em> = 0.0009 (<em>d<\/em> = -0.12); and for the non-parametric fans, the Wilcoxon rank-sum (Mann-Whitney) test yields <em>p<\/em> = 0.0001.<\/p>\n<p>Using <a href=\"http:\/\/wayback.archive.org\/web\/20121019073306\/http:\/\/www.amazon.com\/The-Fault-Stars-John-Green\/dp\/0525478817\">2012-10-19<\/a> as dividing date, yields similar results:<\/p>\n<ul>\n<li>Before (1051 ratings): mean = 4.77 (<abbr title=\"standard deviation\">SD<\/abbr> = 0.020)<\/li>\n<li>After (2291 ratings): mean = 4.71 (<abbr title=\"standard deviation\">SD<\/abbr> = 0.015)<\/li>\n<\/ul>\n<p>A <em>t<\/em>-test (two-sided, unequal variances) yields <em>p<\/em> = 0.0188 (<em>d<\/em> = -0.09); the Wilcoxon rank-sum test yields <em>p<\/em> = 0.0008. Of course, significance testing might be a questionable procedure in this case &#8211; and also in general.<\/p>\n<p>This is actually a census of all Amazon ratings, so there&#8217;s no need to test whether ratings differ. The sample is the population. However, the written reviews could be regarded as a subsample of the ratings of all readers.<\/p>\n<p>Is it a random sample? I don&#8217;t think so. So can we draw proper conclusions from the significance test results? Nah. I won&#8217;t provide a comprehensive discussion of the benefits and problems associated with the null hypothesis significance testing (<a title=\"One of my favourite introductory articles on the topic: Cohen, J. (1994). The earth is round (p&lt;. 05). American Psychologist, 49, 997-1003.\" href=\"http:\/\/dx.doi.org\/10.1037\/0003-066X.49.12.997\" target=\"_blank\">NHST<\/a>). I&#8217;ll just name one of my favourite objections, which Cohen (<a title=\"Cohen, J. (1990). Things I have learned (so far). American Psychologist , 45 , 1304-1312\" href=\"http:\/\/dx.doi.org\/10.1037\/0003-066X.45.12.1304\" target=\"_blank\">1990<\/a>, p. 1308) phrased nicely: &#8220;The null hypothesis, taken literally (and that&#8217;s the only way you can take it in formal hypothesis testing), is <em>always<\/em> false in the real world.&#8221; In the present, the null hypothesis would mean, that average rating of newer readers is exactly the same as the average rating of those who pre-ordered the book etc.<\/p>\n<p>Anyway, the effect size suggests that the drop in ratings is very small, so it should be safe to argue that the book keeps appealing to new readers.<\/p>\n<p>PS: Sorry for nitpicking; this should in no way diminish the article, which I think is highly insightful.<\/p>\n<p>PPS: I spend a good 15 minutes in <a title=\"R\" href=\"http:\/\/www.r-project.org\" target=\"_blank\">R<\/a> trying to beat the data into shape, but I feel much more comfortable in <a title=\"Stata\" href=\"http:\/\/www.stata.com\" target=\"_blank\">Stata<\/a>, so I switched and had the analysis in a few minutes. Here&#8217;s the do-file in case anyone in curious. (Haha, as if!)<\/p>\n<p><!--more--><br \/>\n<code><br \/>\n* written by Johannes Schult (jutze@jutze.com)<br \/>\n* last updated 2013-05-29<br \/>\n*<br \/>\n* A quick significance test.<br \/>\n* H_0: The average rating of The Fault in Our Stars is the same before and after 2013-02-03<br \/>\n*<br \/>\n* Time 1 ratings are from 2013-02-03: http:\/\/wayback.archive.org\/web\/20130203060552\/http:\/\/www.amazon.com\/The-Fault-Stars-John-Green\/dp\/0525478817<br \/>\n* Time 2 ratings are from 2013-05-29: http:\/\/www.amazon.com\/dp\/0525478817\/<br \/>\n*<br \/>\nversion 12.1<br \/>\nset more off<br \/>\nclear<br \/>\nset obs 10<br \/>\ngenerate var1 = 5 in 1<br \/>\nreplace var1 = 4 in 2<br \/>\nreplace var1 = 3 in 3<br \/>\nreplace var1 = 2 in 4<br \/>\nreplace var1 = 1 in 5<br \/>\nreplace var1 = 5 in 6<br \/>\nreplace var1 = 4 in 7<br \/>\nreplace var1 = 3 in 8<br \/>\nreplace var1 = 2 in 9<br \/>\nreplace var1 = 1 in 10<br \/>\ngenerate var2 = 1 in 1\/5<br \/>\nreplace var2 = 2 in 6\/10<br \/>\ngenerate var3 = 1772 in 1<br \/>\nreplace var3 = 226 in 2<br \/>\nreplace var3 = 65 in 3<br \/>\nreplace var3 = 33 in 4<br \/>\nreplace var3 = 14 in 5<br \/>\nreplace var3 = (2743-1772) in 6<br \/>\nreplace var3 = (393-226) in 7<br \/>\nreplace var3 = (122-65) in 8<br \/>\nreplace var3 = (60-33) in 9<br \/>\nreplace var3 = (24-14) in 10<br \/>\n* uncomment the next ten lines to use alternative split time point 2012-10-19:<br \/>\n* http:\/\/wayback.archive.org\/web\/20121019073306\/http:\/\/www.amazon.com\/The-Fault-Stars-John-Green\/dp\/0525478817<br \/>\n*replace var3 = 899 in 1<br \/>\n*replace var3 = 91 in 2<br \/>\n*replace var3 = 35 in 3<br \/>\n*replace var3 = 20 in 4<br \/>\n*replace var3 = 6 in 5<br \/>\n*replace var3 = (2743-899) in 6<br \/>\n*replace var3 = (393-91) in 7<br \/>\n*replace var3 = (122-35) in 8<br \/>\n*replace var3 = (60-20) in 9<br \/>\n*replace var3 = (24-6) in 10<br \/>\nrename var1 rating<br \/>\nrename var2 time<br \/>\nrename var3 amount<br \/>\nexpand amount<br \/>\ndrop amount<br \/>\ncompress<br \/>\noneway rating time, tabulate<br \/>\nttest rating, by(time) unequal<br \/>\n* effect size: Cohen's d<br \/>\ndi -(r(mu_1)-r(mu_2))\/(sqrt(((r(N_1)-1)*r(sd_1)^2+(r(N_2)-1)*r(sd_2)^2)\/(r(N_1)+r(N_2))))<br \/>\nranksum rating, by(time)<br \/>\nclear<br \/>\nexit<br \/>\n* DFTBA!<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"<p>fishingboatproceeds [&#8230;] The same is true on amazon, where the book\u2019s average rating has actually gone up a bit in the past six months (although not in a statistically significant way). [&#8230;] Actually, the ratings have decreased in a statistically significant way (alpha &lt; .05). I used the two most recently archived pages from archive.org, [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,16],"tags":[511,512,510,130,496],"class_list":["post-1325","post","type-post","status-publish","format-standard","hentry","category-books","category-data-analysis","tag-nhst","tag-rating","tag-significance","tag-stata","tag-the-fault-in-our-stars"],"_links":{"self":[{"href":"https:\/\/www.jutze.com\/index.php?rest_route=\/wp\/v2\/posts\/1325","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.jutze.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.jutze.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.jutze.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.jutze.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1325"}],"version-history":[{"count":3,"href":"https:\/\/www.jutze.com\/index.php?rest_route=\/wp\/v2\/posts\/1325\/revisions"}],"predecessor-version":[{"id":1328,"href":"https:\/\/www.jutze.com\/index.php?rest_route=\/wp\/v2\/posts\/1325\/revisions\/1328"}],"wp:attachment":[{"href":"https:\/\/www.jutze.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.jutze.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.jutze.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}