While working on my Boggle project, I realized that I needed a reputable word list to compare submitted words against. Many of the word lists I had found online didn't seem trustworthy, so I thought to use the official word list for scrabble. After searching for a bit and having no luck finding a raw text word list, I found the Official Merriam Scrabble Word Finder. On this site, they allow you to enter a word to check if it's in the official scrabble dictionary. Obviously that wasn't going to be helpful to me unless I made an HTTP request to the site each time a user submitted a word.
Luckily for me, they allow you to also to search for words beginning with some prefix. If you enter a single letter, they will provide a list of all the words beginning with that letter. As long as I could enter each of the 26 letters in the English alphabet into the search bar and scrape the resulting pages, I was going to be home free.
Only a small hurdle stood in my way. If I downloaded the page I was returned immediately, it would only return a little under 1000 words for "a" after passing it through my scraping script even though there were clearly more than that many words starting with "a". Why was that? It turns out that the page would split up the words by their length, and if there were more than 75 words for a certain length, they would require you to press a button to show all the words. I wrote a couple lines of Javascript to grab all of those buttons and simulate a click on them. After that, I was able to go to each of the 26 "starts with" pages, download them, and pass them into my scraping script to get a raw text word list.
If you want the word list or to view the source code, you can find those here.