Jekyll2021-01-14T20:36:00+00:00https://tylerlewiscook.github.io/feed.xmlApproximately NormalDescription goes here.Tyler CookeBay Scam?2020-01-11T00:00:00+00:002020-01-11T00:00:00+00:00https://tylerlewiscook.github.io/ebay-scam<p>Last night I fell down the YouTube rabbit hole and ended up watching <a href="https://youtu.be/A_3L5l2br-4">this</a> video. It shows a coin collector who purchased 5 “random” grab bags from an eBay seller. The listing claims the buyer gets a random coin from a lot of 74 total coins. Also, of the 74 coins, 43 of them contain 1 oz of silver (these are worth more). It turned out that the YouTuber only ended up with 1 of the more valuable 1 oz silver coins out of the 5 coins he ordered. That outcome seemed unlikely to him so he had some concerns about whether the eBay seller was running a scam.</p>
<p>Curious about that outcome as well, I decided to calculate the probability of getting exactly 1 of the more valuable coins when ordered 5 total grab bags. For simplicity, I will assume the seller hasn’t sold any grab bags yet so the total number of coins available is still 74. The distribution of the number of 1 oz silver coins selected <em>k</em> at a time without replacement is a hypergeomtric distribution. The probability can be calculated in R like so:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dhyper</span><span class="p">(</span><span class="m">1</span><span class="p">,</span><span class="m">43</span><span class="p">,</span><span class="m">31</span><span class="p">,</span><span class="m">5</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## [1] 0.08399124
</code></pre></div></div>
<p>If we didn’t know this was a hypergeomtric distribution, then we could also estimate this probability using a simple simulation:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">set.seed</span><span class="p">(</span><span class="m">111</span><span class="p">)</span><span class="w">
</span><span class="n">coins</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="nf">rep</span><span class="p">(</span><span class="s2">"1oz"</span><span class="p">,</span><span class="m">43</span><span class="p">),</span><span class="w"> </span><span class="nf">rep</span><span class="p">(</span><span class="s2">"not_1oz"</span><span class="p">,</span><span class="m">31</span><span class="p">))</span><span class="w">
</span><span class="n">success</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NULL</span><span class="w">
</span><span class="k">for</span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">1</span><span class="o">:</span><span class="m">10000</span><span class="p">){</span><span class="w">
</span><span class="n">bag</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">sample</span><span class="p">(</span><span class="n">coins</span><span class="p">,</span><span class="w"> </span><span class="m">5</span><span class="p">,</span><span class="w"> </span><span class="n">replace</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kc">FALSE</span><span class="p">)</span><span class="w">
</span><span class="n">success</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">ifelse</span><span class="p">(</span><span class="nf">length</span><span class="p">(</span><span class="n">which</span><span class="p">(</span><span class="n">bag</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="s2">"1oz"</span><span class="p">))</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="m">1</span><span class="p">,</span><span class="w"> </span><span class="m">0</span><span class="p">)</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">mean</span><span class="p">(</span><span class="n">success</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## [1] 0.0853
</code></pre></div></div>
<p>Does an 8% chance indicate a scam?</p>Tyler CookLast night I fell down the YouTube rabbit hole and ended up watching this video. It shows a coin collector who purchased 5 “random” grab bags from an eBay seller. The listing claims the buyer gets a random coin from a lot of 74 total coins. Also, of the 74 coins, 43 of them contain 1 oz of silver (these are worth more). It turned out that the YouTuber only ended up with 1 of the more valuable 1 oz silver coins out of the 5 coins he ordered. That outcome seemed unlikely to him so he had some concerns about whether the eBay seller was running a scam. Curious about that outcome as well, I decided to calculate the probability of getting exactly 1 of the more valuable coins when ordered 5 total grab bags. For simplicity, I will assume the seller hasn’t sold any grab bags yet so the total number of coins available is still 74. The distribution of the number of 1 oz silver coins selected k at a time without replacement is a hypergeomtric distribution. The probability can be calculated in R like so: dhyper(1,43,31,5) ## [1] 0.08399124 If we didn’t know this was a hypergeomtric distribution, then we could also estimate this probability using a simple simulation: set.seed(111) coins <- c(rep("1oz",43), rep("not_1oz",31)) success <- NULL for(i in 1:10000){ bag <- sample(coins, 5, replace = FALSE) success[i] <- ifelse(length(which(bag == "1oz")) == 1, 1, 0) } mean(success) ## [1] 0.0853 Does an 8% chance indicate a scam?forestFloor Visualization2019-06-13T00:00:00+00:002019-06-13T00:00:00+00:00https://tylerlewiscook.github.io/forest-floor<p>A collaborator and I recently sumbitted a manuscript analyzing metabolomic data. Part of the analysis involed building a random forest model for classifying patient disease status using metabolite concentrations. The journal we submitted to required an image for a graphical table of contents. I decided this would be a good opportunity to show off the cool things that can be done using the <a href="http://forestfloor.dk/">forestFloor</a> package in R.</p>
<p><img src="https://i.imgur.com/97J9Q2K.png" alt="Imgur" /></p>Tyler CookA collaborator and I recently sumbitted a manuscript analyzing metabolomic data. Part of the analysis involed building a random forest model for classifying patient disease status using metabolite concentrations. The journal we submitted to required an image for a graphical table of contents. I decided this would be a good opportunity to show off the cool things that can be done using the forestFloor package in R.Fantasy Football Player Rankings2018-11-22T00:00:00+00:002018-11-22T00:00:00+00:00https://tylerlewiscook.github.io/fantasy-football-player-rankings<p>It’s Thanksgiving, and for many, that means sitting around with the family watching football after eating lots of turkey. It also means the fantasy football season is just a few weeks away from entering the playoffs. In previous years, that wouldn’t be something that I had much interest in because my fantasy teams have been terrible. This year, however, I am doing…mediocre. Despite the fact that I am currently poised to make the playoffs for once, I don’t feel like my draft or management styles have changed any since finishing last and then second to last in the two prior seasons. I also feel like I had drafted my teams the “right” way according to the experts, and that really made me wonder about the accuracy of fantasy football preseason player rankings.</p>
<p>With that in mind, at the start of this fantasy football season I decided to take a look at how good of a job the professional prognosticators do a predicting fantasy football performance. I found ESPN’s 2017 preseason fantasy player rankings which included opinions from five of their experts. I elected to use the PPR rankings since our league started to use that scoring system this year. For results and the end of the season, I took data from FantasyPros, instead of ESPN, because it was easier to access their data and the points and rankings seemed to match on both sites.</p>
<p>I had to perform a little data cleaning which mostly centered around how the sites handled apostrophes in names differently. I also decided to remove defneses and players who did not play in five or more games becuase I do not think that anyone’s rankings should be punished by injuries. The same should be said for all the poor people who drafted David Johnson with the #1 overall pick last year (like me).</p>
<p>Once that was done, I was ready to proceed with the analysis. I mainly wanted to examine the correlation between preseason and end-of-season player rankings. In order to do that, I calculated Spearman’s correlation and also created a scatterplot to visualize any relationship. The correlation between the two ranks was about 0.38, and you can see what the scatterplot looked like below. Note that I switched up both axes so higher ranked and better performing players should be in the top right.</p>
<p><img src="https://i.imgur.com/HIVgi3T.png" alt="Imgur" /></p>
<p>The correlation is not very strong, and the scatterplot seems to back that up. There are a number of players who were significantly under-ranked as well as several over-ranked (some due to injury and some not). I think the main takeaway is that projecting individual performance in a sport like football is hard, and as an extremely unsuccessful fantasy owner, that makes me feel a little better.</p>
<p>As usual, the code and data can be found on <a href="https://github.com/tylerlewiscook/fantasy-football">my Github</a>.</p>Tyler CookIt’s Thanksgiving, and for many, that means sitting around with the family watching football after eating lots of turkey. It also means the fantasy football season is just a few weeks away from entering the playoffs. In previous years, that wouldn’t be something that I had much interest in because my fantasy teams have been terrible. This year, however, I am doing…mediocre. Despite the fact that I am currently poised to make the playoffs for once, I don’t feel like my draft or management styles have changed any since finishing last and then second to last in the two prior seasons. I also feel like I had drafted my teams the “right” way according to the experts, and that really made me wonder about the accuracy of fantasy football preseason player rankings.Sgt. Pepper’s Word Cloud2018-08-05T00:00:00+00:002018-08-05T00:00:00+00:00https://tylerlewiscook.github.io/sgt-peppers-word-cloud<p>As my last post hinted at, I’ve been working on scraping Beatles lyrics from the web. I was never really able to fully automate the process since the links didn’t have an easily repeatable pattern - the lyrics for each song were on a page with that song in the address. Nevertheless, I was able to complete the collection of lyrics from the “core” albums without too much hassle. Everything can be found in my <a href="https://github.com/tylerlewiscook/beatles-lyrics">beatles-lyrics</a> repo on Github.</p>
<p>I have several analyses planned using this data. To get things started, I thought I’d create a simple word cloud for <em>Sgt. Pepper’s Lonely Hearts Club Band</em>. The code is really simple and I’m using the <code class="language-plaintext highlighter-rouge">tm</code> and <code class="language-plaintext highlighter-rouge">wordcloud</code> packages. I have the songs for each album in their own folder, and all of the album folders are in a folder called “Beatles” that is in my usual R working directory.</p>
<p>First, load the packages and set the working directory:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">library</span><span class="p">(</span><span class="n">tm</span><span class="p">)</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">wordcloud</span><span class="p">)</span><span class="w">
</span><span class="n">setwd</span><span class="p">(</span><span class="s2">"Beatles/"</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<p>Next, load the data and create a corpus from the <em>Sgt. Pepper’s</em> lyrics before doing some basic processing:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sgtp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">VCorpus</span><span class="p">(</span><span class="n">DirSource</span><span class="p">(</span><span class="s2">"SgtPeppers/"</span><span class="p">))</span><span class="w">
</span><span class="n">sgtp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">tm_map</span><span class="p">(</span><span class="n">sgtp</span><span class="p">,</span><span class="w"> </span><span class="n">removePunctuation</span><span class="p">)</span><span class="w">
</span><span class="n">sgtp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">tm_map</span><span class="p">(</span><span class="n">sgtp</span><span class="p">,</span><span class="w"> </span><span class="n">content_transformer</span><span class="p">(</span><span class="n">tolower</span><span class="p">))</span><span class="w">
</span><span class="n">sgtp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">tm_map</span><span class="p">(</span><span class="n">sgtp</span><span class="p">,</span><span class="w"> </span><span class="n">removeNumbers</span><span class="p">)</span><span class="w">
</span><span class="n">sgtp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">tm_map</span><span class="p">(</span><span class="n">sgtp</span><span class="p">,</span><span class="w"> </span><span class="n">stripWhitespace</span><span class="p">)</span><span class="w">
</span><span class="n">sgtp</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">tm_map</span><span class="p">(</span><span class="n">sgtp</span><span class="p">,</span><span class="w"> </span><span class="n">removeWords</span><span class="p">,</span><span class="w"> </span><span class="n">stopwords</span><span class="p">(</span><span class="s2">"english"</span><span class="p">))</span><span class="w">
</span></code></pre></div></div>
<p>I did some transformations like: removing punctuation, removing any numbers, and converting to lower case. Importantly, I also removed “stop words” (words like “the”), but I did not do any stemming.</p>
<p>From there, I created the document-term matix, and only a few more lines of code were needed to produce the word cloud. Note that I am only including words with a minimum frequency of five.</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dtm</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">DocumentTermMatrix</span><span class="p">(</span><span class="n">sgtp</span><span class="p">)</span><span class="w">
</span><span class="n">freq</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">colSums</span><span class="p">(</span><span class="n">as.matrix</span><span class="p">(</span><span class="n">dtm</span><span class="p">))</span><span class="w">
</span><span class="n">set.seed</span><span class="p">(</span><span class="m">1</span><span class="p">)</span><span class="w">
</span><span class="n">wordcloud</span><span class="p">(</span><span class="nf">names</span><span class="p">(</span><span class="n">freq</span><span class="p">),</span><span class="w"> </span><span class="n">freq</span><span class="p">,</span><span class="w"> </span><span class="n">min.freq</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">5</span><span class="p">,</span><span class="w"> </span><span class="n">colors</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">brewer.pal</span><span class="p">(</span><span class="m">3</span><span class="p">,</span><span class="w"> </span><span class="s2">"Accent"</span><span class="p">))</span><span class="w">
</span></code></pre></div></div>
<p>The final result can be seen <a href="https://i.imgur.com/oVXNNpZ.png">here</a>:
<img src="https://i.imgur.com/oVXNNpZ.png" alt="Imgur" /></p>
<p>There should be some more cool stuff coming from this. I know I want to do a sentiment analysis, and I also plan on trying out the <code class="language-plaintext highlighter-rouge">tidytext</code> package. Also, I already have at least one Shiny app planned for this data. Stay tuned!</p>Tyler CookAs my last post hinted at, I’ve been working on scraping Beatles lyrics from the web. I was never really able to fully automate the process since the links didn’t have an easily repeatable pattern - the lyrics for each song were on a page with that song in the address. Nevertheless, I was able to complete the collection of lyrics from the “core” albums without too much hassle. Everything can be found in my beatles-lyrics repo on Github.Quirky rvest Behavior2018-07-25T00:00:00+00:002018-07-25T00:00:00+00:00https://tylerlewiscook.github.io/quirky-rvest-behavior<p>I’ve recently started a new project which involves scraping lyrics from
the web. As before with the World Cup data, I turned to <code class="language-plaintext highlighter-rouge">rvest</code> to
obtain the data. However, this time I noticed some weird behavior with
the results.</p>
<p>Here is what happened:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="n">library</span><span class="p">(</span><span class="n">rvest</span><span class="p">)</span><span class="w">
</span><span class="n">ppm</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">read_html</span><span class="p">(</span><span class="s2">"http://lyrics.wikia.com/wiki/The_Beatles:Please_Please_Me"</span><span class="p">)</span><span class="w">
</span><span class="n">ppm</span><span class="w"> </span><span class="o">%>%</span><span class="w"> </span><span class="n">html_node</span><span class="p">(</span><span class="s1">'.lyricbox'</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w"> </span><span class="n">html_text</span><span class="p">()</span><span class="w">
</span><span class="c1">## [1] "Last night I said these words to my girlI know you never even try, girlCome on (come on), come on (come on)Come on (come on), come on (come on)Please please me, whoa, yeahLike I please youYou don't need me to show the way, loveWhy do I always have to say \"love\"?Come on (come on), come on (come on)Come on (come on), come on (come on)Please please me, whoa, yeahLike I please youI don't wanna sound complainingBut you know there's always rain in my heart (in my heart)I do all the pleasing with youIt's so hard to reason With you, whoa yeahWhy do you make me blue?Last night I said these words to my girlI know you never even try, girlCome on (come on), come on (come on)Come on (come on), come on (come on)Please please me, whoa, yeahLike I please you(Please) me, whoa, yeahLike I please you(Please) me, whoa, yeahLike I please you\n"</span><span class="w">
</span></code></pre></div></div>
<p>As you can see, some of the words are concatenated. This will cause
serious problems for a text analysis since “girlI” is not an actual
word. After going back to inspect the source, it seems like this occurs
whenever there is a new line.</p>
<p>I did some Googling to see if there was an easy way to handle this.
Apparently, a few other people have encountered this same issue and
posted about it on <a href="https://github.com/hadley/rvest/issues/175">Github</a>,
but there doesn’t appear to be a fix posted as well.</p>
<p>It seems like this should be a simple matter to fix, but handling
strings is one of my weaker programming skills. Therefore, I turned to
the r/rstats subreddit for help, and fortunately
<a href="https://www.reddit.com/r/rstats/comments/91ap8q/line_breaks_with_rvest/e2xjmfk">u/Schrodingers-Human</a>
posted a nice solution that seems to work!</p>
<p>Here is my slightly modified solution taken from Schrodingers-Human:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="n">library</span><span class="p">(</span><span class="n">rvest</span><span class="p">)</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">stringr</span><span class="p">)</span><span class="w">
</span><span class="n">ppm</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">read_html</span><span class="p">(</span><span class="s2">"http://lyrics.wikia.com/wiki/The_Beatles:Please_Please_Me"</span><span class="p">)</span><span class="w">
</span><span class="n">ppm</span><span class="w"> </span><span class="o">%>%</span><span class="w">
</span><span class="n">html_node</span><span class="p">(</span><span class="s1">'.lyricbox'</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w">
</span><span class="nf">as.character</span><span class="p">()</span><span class="w"> </span><span class="o">%>%</span><span class="w">
</span><span class="n">str_sub</span><span class="p">(</span><span class="n">start</span><span class="o">=</span><span class="m">23</span><span class="p">,</span><span class="w"> </span><span class="n">end</span><span class="o">=</span><span class="m">-39</span><span class="p">)</span><span class="w"> </span><span class="o">%>%</span><span class="w">
</span><span class="n">str_replace_all</span><span class="p">(</span><span class="s2">"<br>"</span><span class="p">,</span><span class="w"> </span><span class="s2">" "</span><span class="p">)</span><span class="w">
</span><span class="c1">## [1] "Last night I said these words to my girl I know you never even try, girl Come on (come on), come on (come on) Come on (come on), come on (come on) Please please me, whoa, yeah Like I please you You don't need me to show the way, love Why do I always have to say \"love\"? Come on (come on), come on (come on) Come on (come on), come on (come on) Please please me, whoa, yeah Like I please you I don't wanna sound complaining But you know there's always rain in my heart (in my heart) I do all the pleasing with you It's so hard to reason With you, whoa yeah Why do you make me blue? Last night I said these words to my girl I know you never even try, girl Come on (come on), come on (come on) Come on (come on), come on (come on) Please please me, whoa, yeah Like I please you (Please) me, whoa, yeah Like I please you (Please) me, whoa, yeah Like I please you"</span><span class="w">
</span></code></pre></div></div>
<p>Looks much better.</p>Tyler CookI’ve recently started a new project which involves scraping lyrics from the web. As before with the World Cup data, I turned to rvest to obtain the data. However, this time I noticed some weird behavior with the results. Here is what happened: library(rvest) ppm <- read_html("http://lyrics.wikia.com/wiki/The_Beatles:Please_Please_Me") ppm %>% html_node('.lyricbox') %>% html_text() ## [1] "Last night I said these words to my girlI know you never even try, girlCome on (come on), come on (come on)Come on (come on), come on (come on)Please please me, whoa, yeahLike I please youYou don't need me to show the way, loveWhy do I always have to say \"love\"?Come on (come on), come on (come on)Come on (come on), come on (come on)Please please me, whoa, yeahLike I please youI don't wanna sound complainingBut you know there's always rain in my heart (in my heart)I do all the pleasing with youIt's so hard to reason With you, whoa yeahWhy do you make me blue?Last night I said these words to my girlI know you never even try, girlCome on (come on), come on (come on)Come on (come on), come on (come on)Please please me, whoa, yeahLike I please you(Please) me, whoa, yeahLike I please you(Please) me, whoa, yeahLike I please you\n" As you can see, some of the words are concatenated. This will cause serious problems for a text analysis since “girlI” is not an actual word. After going back to inspect the source, it seems like this occurs whenever there is a new line. I did some Googling to see if there was an easy way to handle this. Apparently, a few other people have encountered this same issue and posted about it on Github, but there doesn’t appear to be a fix posted as well. It seems like this should be a simple matter to fix, but handling strings is one of my weaker programming skills. Therefore, I turned to the r/rstats subreddit for help, and fortunately u/Schrodingers-Human posted a nice solution that seems to work! Here is my slightly modified solution taken from Schrodingers-Human: library(rvest) library(stringr) ppm <- read_html("http://lyrics.wikia.com/wiki/The_Beatles:Please_Please_Me") ppm %>% html_node('.lyricbox') %>% as.character() %>% str_sub(start=23, end=-39) %>% str_replace_all("<br>", " ") ## [1] "Last night I said these words to my girl I know you never even try, girl Come on (come on), come on (come on) Come on (come on), come on (come on) Please please me, whoa, yeah Like I please you You don't need me to show the way, love Why do I always have to say \"love\"? Come on (come on), come on (come on) Come on (come on), come on (come on) Please please me, whoa, yeah Like I please you I don't wanna sound complaining But you know there's always rain in my heart (in my heart) I do all the pleasing with you It's so hard to reason With you, whoa yeah Why do you make me blue? Last night I said these words to my girl I know you never even try, girl Come on (come on), come on (come on) Come on (come on), come on (come on) Please please me, whoa, yeah Like I please you (Please) me, whoa, yeah Like I please you (Please) me, whoa, yeah Like I please you" Looks much better.World Cup Shiny App2018-07-23T00:00:00+00:002018-07-23T00:00:00+00:00https://tylerlewiscook.github.io/world-cup-shiny-app<p>The data for World Cup goal times caused me a bit of a headache, and I was happy to get everything sorted out so I could start exploring something new. Nevertheless, an idea for one last visualization using this data has been stuck in my head.</p>
<p>My favorite plot to come out of the original analysis was the distribution of goal times broken down by year. However, there were so many years that I was never able to get the dimensions of the plot to look quite right. The histograms always seemed distorted in some way. To fix this, I decided to try an interactive Shiny app where one could select individual years to visualize.</p>
<p>The visualization I have in mind is really simple and shouldn’t take too much coding. And yes, that is exactly what I <em>incorrectly</em> assumed when I initially scraped the data. Fortunately, things went better this time.</p>
<p>First off, I need to load the data and necessary packages. Since this app is going to be very simple, two packages will do:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">library</span><span class="p">(</span><span class="n">shiny</span><span class="p">)</span><span class="w">
</span><span class="n">library</span><span class="p">(</span><span class="n">ggplot2</span><span class="p">)</span><span class="w">
</span><span class="c1"># data frame is called new.goals</span><span class="w">
</span><span class="n">load</span><span class="p">(</span><span class="s2">"updated_goals.RData"</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<p>Shiny apps have two main pieces: the user interface defintions in <code class="language-plaintext highlighter-rouge">ui</code> and the <code class="language-plaintext highlighter-rouge">server</code> function. I want my interface to have a title panel, year input selection on the side, and plot in the main panel. I can get that with:</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">ui</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">fluidPage</span><span class="p">(</span><span class="w">
</span><span class="n">headerPanel</span><span class="p">(</span><span class="s2">"When World Cup Goals Are Scored"</span><span class="p">),</span><span class="w">
</span><span class="n">sidebarPanel</span><span class="p">(</span><span class="w">
</span><span class="n">selectInput</span><span class="p">(</span><span class="s2">"var"</span><span class="p">,</span><span class="w"> </span><span class="n">label</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"Select year:"</span><span class="p">,</span><span class="w"> </span><span class="n">unique</span><span class="p">(</span><span class="n">new.goals</span><span class="o">$</span><span class="n">year</span><span class="p">))</span><span class="w">
</span><span class="p">),</span><span class="w">
</span><span class="n">mainPanel</span><span class="p">(</span><span class="w">
</span><span class="n">plotOutput</span><span class="p">(</span><span class="s2">"hist"</span><span class="p">)</span><span class="w">
</span><span class="p">)</span><span class="w">
</span><span class="p">)</span><span class="w">
</span></code></pre></div></div>
<p>Then, my server function takes the user-selected year to subset the data to produce a histrogram in <code class="language-plaintext highlighter-rouge">ggplot2</code>.</p>
<div class="language-r highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">server</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="k">function</span><span class="p">(</span><span class="n">input</span><span class="p">,</span><span class="w"> </span><span class="n">output</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">selectedYear</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">reactive</span><span class="p">({</span><span class="w">
</span><span class="n">subset</span><span class="p">(</span><span class="n">new.goals</span><span class="p">,</span><span class="w"> </span><span class="n">year</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">input</span><span class="o">$</span><span class="n">var</span><span class="p">)</span><span class="w">
</span><span class="p">})</span><span class="w">
</span><span class="n">output</span><span class="o">$</span><span class="n">hist</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">renderPlot</span><span class="p">({</span><span class="w">
</span><span class="n">ggplot</span><span class="p">(</span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">selectedYear</span><span class="p">(),</span><span class="w"> </span><span class="n">aes</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">time</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">geom_histogram</span><span class="p">(</span><span class="n">breaks</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">seq</span><span class="p">(</span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="m">125</span><span class="p">,</span><span class="w"> </span><span class="m">5</span><span class="p">),</span><span class="w"> </span><span class="n">color</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'black'</span><span class="p">,</span><span class="w"> </span><span class="n">fill</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'steelblue3'</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w">
</span><span class="n">labs</span><span class="p">(</span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Game Time'</span><span class="p">,</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Count'</span><span class="p">)</span><span class="w">
</span><span class="p">})</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Finally, running <code class="language-plaintext highlighter-rouge">shinyApp(ui = ui, server = server)</code> will launch the app locally.</p>
<p>This app is currently up and running on my <a href="https://tylerlewiscook.shinyapps.io/WorldCupGoals/">shinyapps.io</a> page. Shiny is super cool, and a little different than normal R use, so I imagine more Shiny posts will come in the future.</p>Tyler CookThe data for World Cup goal times caused me a bit of a headache, and I was happy to get everything sorted out so I could start exploring something new. Nevertheless, an idea for one last visualization using this data has been stuck in my head.Visualizing When World Cup Goals Are Scored2018-07-17T00:00:00+00:002018-07-17T00:00:00+00:00https://tylerlewiscook.github.io/visualizing-world-cup-goals<p>A surprising amount of hard work went into obtaining the simple dataset containing the game times when World Cup goals are scored, and now it is time to see if that work will pay off. Fortunately, the actual analysis I want to do is very simple and it shouldn’t take much effort using <code class="language-plaintext highlighter-rouge">ggplot2</code>.</p>
<p>Before plotting, a little data cleaning/processing is needed. The raw data has each time as a string, and stoppage time goals look something like: 45’+2’. I saw a few optoins for handling the stoppage time goals. First, I could treat the goal times more like categorical data and just have something like a “45+” group. This, however, loses a little bit of information since one can’t see exactly when the goals are scored. My next thought was to just add the times together. So 45’+2’ would become 47’. The problem with this is that a goal scored in the second minute of stoppage time during the first half and a goal scored 2 minutes into the second half would be treated the same. I ended up going with the latter option and chose to create a new variable to indicate whether a goal was scored in the first half, second half, or extra time. This would help fix the problem created by adding up the stoppage time goals. I imagine there are several more ways to address this issue, and some might prove superior to what I chose to do, but this seemed like a good solution given I just wanted to get a quick sense of the what is going on with the data.</p>
<p>Below is the first plot showing a histogram of all of the times when goals are scored:
<img src="https://i.imgur.com/IEjNDG6.png" alt="Imgur" />
The distribution does not quite seem to be uniform. Interestingly, to me it looks like there is a little spike towards the end of the second half. I sort of expected this as teams might be motivated to press harder to get a goal to win or tie the game.</p>
<p>Next, I created a plot looking at the goal times broken down by “half” (so first, second, or extra time). Those histograms are below:
<img src="https://i.imgur.com/x3FcqjW.png" alt="Imgur" /></p>
<p>Finally, I wanted to see if the distribution looked different for different World Cups. I created separate histograms for each year in order to investigate that question. Those series of plots are here:
<img src="https://i.imgur.com/tRO9GIR.png" alt="Imgur" />
The distribution seems for uniform in some years, and some years have some unique characteristics. For example, what happened to boost the scoring right around half time in 1998?</p>
<p>The cleaned data with corresponding code, as well as the code used to create all of the histograms (with a couple bonus plots), can be found on <a href="https://github.com/tylerlewiscook/world-cup">my Github</a>.</p>
<p>P.S. At some point during all of this I found an excellent World Cup dataset on Kaggle that includes some other useful information. You can check that out <a href="https://www.kaggle.com/abecklas/fifa-world-cup">here</a>.</p>Tyler CookA surprising amount of hard work went into obtaining the simple dataset containing the game times when World Cup goals are scored, and now it is time to see if that work will pay off. Fortunately, the actual analysis I want to do is very simple and it shouldn’t take much effort using ggplot2.Scraping World Cup Goal Data2018-07-03T00:00:00+00:002018-07-03T00:00:00+00:00https://tylerlewiscook.github.io/scraping-world-cup-data<p>Watching the World Cup over the last couple of weeks has got me wondering several things. Is the stoppage time determination completely arbitrary? What is with that magical medical spray? Will Neymar ever walk the same after that <a href="https://www.reddit.com/r/soccer/comments/8vjhtz/neymar_rolling_around_in_pain_after_getting/">vicious injury</a>?</p>
<p>Several of these questions are difficult to answer, but one potential thing I could investigate is the distribution of when goals are scored during a game. I’m interested in things like comparing the frequency of goals scored in each half and seeing if there is an increase in goals near the end of the game. As far as the statistics go, my curiosity could be satisfied with some simple plots so finding the right data is probably the only challenging thing I will need to do.</p>
<p>A quick Google search did not yield the exact data that I needed so I would have to compile everything myself. This wasn’t a deterrence, however, because it presented an excellent opportunity to refresh my web scraping skills in R. I decided to scrape the data from Wikipedia using the <a href="https://github.com/hadley/rvest">rvest</a> package in coordination with <a href="https://selectorgadget.com/">SelectorGadget</a>.</p>
<p>Here is the bit of code that got me started:</p>
<div class="language-R highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">library</span><span class="p">(</span><span class="n">rvest</span><span class="p">)</span><span class="w">
</span><span class="n">years</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">seq</span><span class="p">(</span><span class="m">1930</span><span class="p">,</span><span class="w"> </span><span class="m">2014</span><span class="p">,</span><span class="w"> </span><span class="m">4</span><span class="p">)</span><span class="w">
</span><span class="n">years</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">years</span><span class="p">[</span><span class="o">-</span><span class="nf">c</span><span class="p">(</span><span class="n">which</span><span class="p">(</span><span class="n">years</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1942</span><span class="p">),</span><span class="w"> </span><span class="n">which</span><span class="p">(</span><span class="n">years</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="m">1942</span><span class="p">)</span><span class="m">+1</span><span class="p">)]</span><span class="w">
</span><span class="n">times</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NULL</span><span class="w">
</span><span class="n">which.cup</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="kc">NULL</span><span class="w">
</span><span class="k">for</span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">1</span><span class="o">:</span><span class="nf">length</span><span class="p">(</span><span class="n">years</span><span class="p">)){</span><span class="w">
</span><span class="n">page</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">paste</span><span class="p">(</span><span class="s2">"https://en.wikipedia.org/wiki/"</span><span class="p">,</span><span class="w"> </span><span class="n">years</span><span class="p">[</span><span class="n">i</span><span class="p">],</span><span class="w"> </span><span class="s2">"_FIFA_World_Cup"</span><span class="p">)</span><span class="w">
</span><span class="n">temp.page</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">read_html</span><span class="p">(</span><span class="n">page</span><span class="p">)</span><span class="w">
</span><span class="n">nodes</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">html_nodes</span><span class="p">(</span><span class="n">temp.page</span><span class="p">,</span><span class="w"> </span><span class="s1">'.mw-parser-output div td small'</span><span class="p">)</span><span class="w">
</span><span class="n">output</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="n">html_text</span><span class="p">(</span><span class="n">nodes</span><span class="p">)</span><span class="w">
</span><span class="n">times</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="n">times</span><span class="p">,</span><span class="w"> </span><span class="n">output</span><span class="p">)</span><span class="w">
</span><span class="n">which.cup</span><span class="w"> </span><span class="o"><-</span><span class="w"> </span><span class="nf">c</span><span class="p">(</span><span class="n">which.cup</span><span class="p">,</span><span class="w"> </span><span class="nf">rep</span><span class="p">(</span><span class="n">years</span><span class="p">[</span><span class="n">i</span><span class="p">],</span><span class="w"> </span><span class="nf">length</span><span class="p">(</span><span class="n">output</span><span class="p">)))</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>At this point, things looked good. I manually checked all the times for the 1930 World Cup and everything matched. Next, I created a table displaying the total number of goals scored for each World Cup. Curiously, my results indicated that there were only 17 total goals scored in 1982, and there was no way that was right. For some reason, starting in the 1970s, the main Wikipedia entries for each World Cup stop displaying the exact times goals are scored during group play. That means I needed to get additional data from each individual group page to combine with what I already had for the knockout portion of each tournament.</p>
<p>That was frustrating, but it seemed like the solution would be fairly straightforward. Unfortunately, here is where things really hit the fan. So today, World Cup groups are identified by letters, but they were previously numbered. The size of the tournament has changed over the years so the number of groups has changed as well. Also, in 1982 there were two rounds of group play!</p>
<p>This inconsistency in the tournament format meant I had to deal with several special cases. I ended up using an inelegant “brute force” approach to writing the code for the remaining group stage data, and I am fairly confident that a more efficient solution is out there (possibly using a different data source). Nevertheless, in the end I had the data I wanted. The raw data from Wikipedia as well as the R script that I used to obtain the data can be found on Github <a href="https://github.com/tylerlewiscook/world-cup">here</a>.</p>Tyler CookWatching the World Cup over the last couple of weeks has got me wondering several things. Is the stoppage time determination completely arbitrary? What is with that magical medical spray? Will Neymar ever walk the same after that vicious injury?That First Blog Post About The Blog2018-06-28T00:00:00+00:002018-06-28T00:00:00+00:00https://tylerlewiscook.github.io/that-first-blog-post-about-the-blog<p>To get things started off, I thought I would explain some of the details about how this site came into existence. I can’t remember coming across any particular site that really served as motivation, but somehow, I read about <a href="https://pages.github.com/">GitHub Pages</a> and Jekyll. I don’t really have any experience with website creation and design so I had to rely on Google and the numerous resources that others have graciously shared online to figure out how to get things to work.</p>
<p>Starting with the basics, I found this YouTube video which appears to be a recording of a workshop from the University of Idaho Library:</p>
<p><a href="https://www.youtube.com/watch?v=SWVjQsvQocA"><img src="https://img.youtube.com/vi/SWVjQsvQocA/0.jpg" alt="Workshop video" /></a></p>
<p>After watching most of that video and looking over the accompanying website, I decided I would take a learn-by-doing approach and start working on customizing my own site. First off, I wanted to pick a slick theme. None of the built-in Jekyll themes available straight form GitHub Pages did too much for me so I did some searching for alternatives. That’s when I came across <a href="https://mmistakes.github.io/minimal-mistakes/">Minimal Mistakes</a> and the corresponding (and super helpful) <a href="https://mmistakes.github.io/minimal-mistakes/docs/quick-start-guide/">Quick-Start Guide</a>. There is a Minimal Mistakes template available on GitHub that I forked and started editing. And that brings us to here…</p>Tyler CookTo get things started off, I thought I would explain some of the details about how this site came into existence. I can’t remember coming across any particular site that really served as motivation, but somehow, I read about GitHub Pages and Jekyll. I don’t really have any experience with website creation and design so I had to rely on Google and the numerous resources that others have graciously shared online to figure out how to get things to work.