" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Hospitalizations from COVID-19 have increased nearly 90% and California officials say they could triple by Christmas. https://t.co/hrBnP04HnB
— KRON4 News (@kron4news) December 1, 2020
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Something for the afternoon slump / journey home / after school / cooking dinner ... a special 30 minute mix of cool Christmas tunes intercut with Christmas film samples and scratching @BBCSounds https://t.co/rHovIA3u5e
— Chris Hawkins (@ChrisHawkinsUK) December 1, 2020
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Algorithm used and text classification" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Supervised training and bag of words method" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With labelled data (supervised training), a model can identify the positive and negative words according to a score. For example, if we want to do opinion mining about a product on Twitter, it is easier if we already have access to some labelled data, for example, a list of customer reviews with a score rating, from another website. By matching the score and the reviews, the model will be able to extract the words that are most often used for positive or negative ratings. This process is known as a \"bag-of-words\" model: each word is treated separately as a single instance, and the model does not process the grammar or the syntax of the sentence. This is an easy way to deal with text analysis.\n", "\n", "Let's understand this with an example (note: this example is imaginative and not reproducible as it is). Here, we took some food reviews together with a score, all written by customers, on the platform Amazon. From this data and the score, the model determined the most common words in _positive_ (4 or 5 scored) and in _negative_ (1 or 2 scored) reviews. Based on those lists of words, the model can now label new unseen reviews into _positive_ or _negative_, depending on which words are present in it. \n", "\n", "\n", "_Data from Kaggle \"Amazon Fine Food Reviews\" and accessible [here](https://www.kaggle.com/snap/amazon-fine-food-reviews?select=Reviews.csv)._\n", "\n", "If the data is available, this step can probably give better results as it is focused on a particular product or brand, giving more targetted results.\n", "\n", "If the data is not available, it is possible to use models that are already trained, with a list of positive or negative words.\n", "\n", "_Note_: this is one possibility to deal with text analysis, but other methods exist, of course." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Tokenization" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tokenization is the process of separating a sentence or a text into smaller parts. Mostly, we want to tokenize the text into words, to be able to analyze each word separately.\n", "\n", "A tokenization method takes a text and gives a list of the words that the text contains." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{toggle} Advanced level\n", "There are several methods to do that.\n", "\n", "The simplest is the built-in method ``split()`` used on a string. It separates a string into words, separated by a space. This, however, does not allow to separate punctuation. Depending on the cleaning process in use, that we detail in the next pages, this can be of importance or not.\n", "\n", "Some libraries have more advanced functions for tokenization. For example, the library ``NLTK`` has the function [word_tokenize](https://www.nltk.org/api/nltk.tokenize.html#nltk.tokenize.punkt.PunktLanguageVars.word_tokenize), which is able to separate punctuation as well.\n", "```" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "tags": [ "hide-input", "hide-output" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than most.\n", "['I', 'have', 'bought', 'several', 'of', 'the', 'Vitality', 'canned', 'dog', 'food', 'products', 'and', 'have', 'found', 'them', 'all', 'to', 'be', 'of', 'good', 'quality.', 'The', 'product', 'looks', 'more', 'like', 'a', 'stew', 'than', 'a', 'processed', 'meat', 'and', 'it', 'smells', 'better.', 'My', 'Labrador', 'is', 'finicky', 'and', 'she', 'appreciates', 'this', 'product', 'better', 'than', 'most.']\n" ] } ], "source": [ "# With the built-in method split()\n", "\n", "review = \"I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than most.\"\n", "print(review)\n", "tokens = review.split()\n", "print(tokens)" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than most.\n", "['I', 'have', 'bought', 'several', 'of', 'the', 'Vitality', 'canned', 'dog', 'food', 'products', 'and', 'have', 'found', 'them', 'all', 'to', 'be', 'of', 'good', 'quality', '.', 'The', 'product', 'looks', 'more', 'like', 'a', 'stew', 'than', 'a', 'processed', 'meat', 'and', 'it', 'smells', 'better', '.', 'My', 'Labrador', 'is', 'finicky', 'and', 'she', 'appreciates', 'this', 'product', 'better', 'than', 'most', '.']\n" ] } ], "source": [ "# With the library nltk\n", "\n", "from nltk import word_tokenize\n", "\n", "review = \"I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than most.\"\n", "print(review)\n", "tokens = word_tokenize(review)\n", "print(tokens)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Go onto the next page to learn how this text can be preprocessed before tokenization, for a better result of the analysis." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Quiz" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "data": { "text/html": [ "\n", "This happened in Adelaide the other day. #koala #Adelaide https://t.co/vAQFkd5r7q
— 893LAFM (@893lafm) December 9, 2020