Project: Text Analyser

Customers leave reviews, and Ava wants to know what they talk about. Which words come up the most? How long are the reviews? Which reviews are positive? This last project builds a small class, that analyses text, and uses strings, sets, counters, and regular expressions. It is also the first step towards real language processing.

• What do customers talk about?
• Count words, lengths and sentiment
• A clean class with small methods

Cleaning and splitting text

The first step with any text is cleaning. Make it lower case, and pull out the words with a regular expression, that keeps letters and apostrophes. Remove stop words, like the and and, which carry little meaning. What remains are the words that matter.

import re

STOP = {"the", "and", "a", "is", "it", "to", "of", "i"}

def words(text):
    found = re.findall(r"[a-z']+", text.lower())
    return [w for w in found if w not in STOP]

print(words("The fern is lovely, and it arrived FAST!"))

['fern', 'lovely', 'arrived', 'fast']

A TextAnalyser class

Now a class. It keeps the cleaned words, and offers methods, for the word count, the most common words, and the average word length. Each method is short. Create it once, and ask many questions. This is how analysis code is organised in real projects.

import re
from collections import Counter

class TextAnalyser:
    STOP = {"the", "and", "a", "is", "it", "to"}

    def __init__(self, text):
        self.words = [w for w in re.findall(r"[a-z']+", text.lower()) if w not in self.STOP]

    def top(self, n=3):
        return Counter(self.words).most_common(n)

    def avg_len(self):
        return round(sum(map(len, self.words)) / len(self.words), 2)

t = TextAnalyser("Lovely fern, lovely pot, and the fern is healthy. Fast delivery!")
print(len(t.words), t.top(2), t.avg_len())

8 [('lovely', 2), ('fern', 2)] 5.25

Simple sentiment

For a taste of sentiment analysis, keep two sets of words, positive and negative. Score each review, by counting the positive words, minus the negative ones. It is crude, but surprisingly useful, for a first pass. Real systems use machine learning, but the idea of scoring is the same.

POS = {"lovely", "healthy", "fast", "great", "happy"}
NEG = {"dead", "late", "broken", "slow"}

def score(review):
    ws = set(review.lower().replace(",", " ").replace("!", " ").split())
    return len(ws & POS) - len(ws & NEG)

for r in ["Lovely plant, fast delivery!", "Arrived late and dead", "It is a pot"]:
    print(score(r), r)

2 Lovely plant, fast delivery!
-2 Arrived late and dead
0 It is a pot

Longest word

Your turn. Write a function that returns the longest word in a sentence. Split the text, and use max with the key set to len. Ready? Here is one solution. Split, then max with a key, and a test.

def longest(text):
    return max(text.split(), key=len)

print(longest("water the monstera daily"))

monstera

Summary