Comments on: Using Python to detect the most frequent words in a file

By: david

david — Fri, 28 Mar 2008 18:14:30 +0000

count = {}; open(”somefile”).each_line { |line| line.split(/\b/).each { |word| count[word] ||= 0; count[word] += 1 } }

By: Mark

Mark — Wed, 26 Mar 2008 03:00:19 +0000

Somehow, my sort line above got mangled. This one’s an improvement:

top_words = count.sort_by { |w| w[1] }

By: William Chang

William Chang — Mon, 24 Mar 2008 07:03:28 +0000

This of course depends on what you are counting words for, but I would recommend translate all non-letters to space and then splitting on space. For the natural language tasks that I do, this is pretty appropriate. It really depends on what you want to happen when you hit stuff like “Bob’s”, “hyper-active”, “http://www.google.com”, “bob@gmail.com”, “2342sdf”, etc… I also like putting the rule that I use to split into words into it’s own function which I call here tokenize().

from string import punctuation
from collections import defaultdict

N = 10
words = {}

def tokenize(line):
line = re.sub(r”[^a-z]”, ” “, line.lower())
return line.split()

words = defaultdict(int)
for line in open(”test.txt”):
for token in tokenize(line):
words[token] +=1

top_words = sorted(words.iteritems(),
key=lambda(word, count): (-count, word))[:N]

for word, frequency in top_words:
print “%s: %d” % (word, frequency)

By: Mark

Mark — Sun, 23 Mar 2008 01:41:48 +0000

I’m learning ruby, so I thought I’d put together a Ruby version:

N = 10
count = Hash.new(0)

File.open(the_file).each_line do |line|
  line.downcase.scan(/\w+/).each do |word|
    count[word] += 1
  end
end

top_words = count.sort{|a,b| a[1]b[1]}

top_words.each do |top|
  puts "%s: %d" % top
end

By: Antonio Cangiano

Antonio Cangiano — Wed, 19 Mar 2008 12:34:56 +0000

Hi Paddy, thanks for your comment. I've slightly changed the wording to point out that in the "good" solution at the end, we are using key rather than cmp. Using defaultdict would work (2.5 only) and also be more efficient. Here is the solution that incorporates your suggestion:

from string import punctuation
from collections import defaultdict

N = 10
words = {}

words_gen = (word.strip(punctuation).lower() for line in open("test.txt")
                                             for word in line.split())

words = defaultdict(int)
for word in words_gen:
    words[word] +=1

top_words = sorted(words.iteritems(),
                   key=lambda(word, count): (-count, word))[:N] 

for word, frequency in top_words:
    print "%s: %d" % (word, frequency)

By: Paddy3118

Paddy3118 — Wed, 19 Mar 2008 05:54:47 +0000

Hi Antonio, Just before the last code section, you introduce it as "getting rid of reverse=True", but you fail to mention that you also change from using cmp to use key. cmp is called for every comparison wheras key is called once for each item in the list which is usually faster.
I also wonder if this code:

for word in words_gen:
    words[word] = words.get(word, 0) + 1

Might be replaced by (untested):

words = defaultdict(int)
for word in words_gen:
    words[word] +=1

Which would look up word in words only once?
I enjoyed your post.
Thanks, Paddy.

By: Antonio Cangiano

Antonio Cangiano — Tue, 18 Mar 2008 13:13:46 +0000

Nice one Tom.

By: tlrobinson.net / blog » Blog Archive » Using command line tools to detect the most frequent words in a file

Tue, 18 Mar 2008 08:30:44 +0000

[…] Cangiano wrote a post about “Using Python to detect the most frequent words in a file“. It’s a nice summary of how to do it in Python, but (nearly) the same thing can be […]

By: Tom Robinson

Tom Robinson — Tue, 18 Mar 2008 07:37:58 +0000

Of course, this is exactly what command line filter programs are good at…