1. Exquisite Tweets from @OrbenAmy

    PreoccupationsCollected by Preoccupations

    Large quantities of data = High quality of science. My new Nature Human Behaviour paper published today with @ShuhBillSkee shows that this could not be further from the truth. nature.com/articles/s4156… (1/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    For our work we focus on the association between digital technology use and teen well-being, which has been getting a lot of press lately. Too much, if you ask me. (2/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    Some days people find negative associations, some days positive, some days none at all. Why is there no consistency in scientific findings? (3/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    Many of these studies examine openly-available large-scale datasets with many thousands of variables and participants. We found that almost every paper analyses such data slightly differently, e.g. in how they define well-being (4/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    But also in how they define digital technology use, what control measures they use and what models they run. We found that scientists could have analysed the data in over a trillion different ways - just to answer 1 simple research question! (5/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    We used Specification Curve Analysis (an innovative statistical technique), to examine how much impact this potential for analytical flexibility has on the scientific conclusions reached in a resulting paper. (6/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    Short answer: A huge impact! Running over 20,000 analyses on 3 different datasets we found that scientists could have written 1000s of papers finding positive, negative and non-significant associations between well-being and digital tech - it all depends on the analysis (7/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    For example, defining digital technology use in different ways affects the results found (8/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    Or… whether you include good quality controls in the analysis or not. As you see, not having proper controls makes the association between digital technology use and well-being a lot more negative. (9/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben

    The paper powerfully visualises that without pre-registering analysis plans beforehand, analytical bias can allow researchers to tell almost any story with powerful data resources. Is this really what science is supposed to be like? (10/10)

    Reply Retweet Like

    OrbenAmy

    Amy Orben