May 2012 – Acodemics

Language Detection Algorithm

My thesis required dealing with large quantities of Twitter data and to make it easier for myself I decided to only use English language tweets. Due to the ‘interesting’ grammar and spelling that is frequently found on line a different algorithm to a standard dictionary test was required. This one checks for common English n-grams (letter groupings) and returns a value of how sure the input is English.

Here’s some examples:

Good morning everyone! : 66%
Im So Sick. Really in a bad Position : 50%
Thats the fear of unicorns : 100%
Gnt too com uma saudade daminha namorada vcs nao tem ideia :(((( : 0%
Ils testent les hologrammes pour faire des concerts par des morts. Bientôt même les pas morts feront des hologrammes et vous irez les voir. : 12%
il7ain ilnass ykhl9un men habat twitter o yntqlun to istagram=))!!! : 40%

from collections import Counter

class trigram:
    ngrams = ['the', 'and', 'tha', 'ent', 'ing', 'ion', 'tio', 'for',
        'nde', 'has', 'nce', 'edt', 'tis', 'oft', 'sth', 'men', ' th', 
        'he ', 'ed ', ' of', 'in ', 'to ', 'er ', 'as ', 'her', 'ng ',
        'of ', ' an', ' in']
    probability = 0.0

    def __init__(self):
        pass

    def parse_text(self, text):
        count = 0.0
        s = set()
        length = len(text.strip(' ').split())
        for i in range( len( text ) ): s.add( text[ i: i + 3 ] )
        c = Counter(s)
        for trigram in self.ngrams:
            try:
                count += c[trigram]
            except IndexError as e:
                pass
        if length > 0:
            self.probability = (count / length) * 100.0
        else:
            self.probability = 0
        return self.probability

    def get_probability(self):
        return self.probability

    def is_english(self):
        return (self.probability > 40)

from collections import Counter

class trigram:

ngrams = ['the', 'and', 'tha', 'ent', 'ing', 'ion', 'tio', 'for',

'nde', 'has', 'nce', 'edt', 'tis', 'oft', 'sth', 'men', ' th',

'he ', 'ed ', ' of', 'in ', 'to ', 'er ', 'as ', 'her', 'ng ',

'of ', ' an', ' in']

probability = 0.0

def __init__(self):

pass

def parse_text(self, text):

count = 0.0

s = set()

length = len(text.strip(' ').split())

for i in range( len( text ) ): s.add( text[ i: i + 3 ] )

c = Counter(s)

for trigram in self.ngrams:

try:

count += c[trigram]

except IndexError as e:

pass

if length > 0:

self.probability = (count / length) * 100.0

else:

self.probability = 0

return self.probability

def get_probability(self):

return self.probability

def is_english(self):

return (self.probability > 40)

Python Feed Forward Perceptron

May 27, 2012 / Jack Concanon / 1 Comment

I thought I would post a couple of useful bits of code from my final year of university.

The first is a feed forward perceptron written in Python. I tried to expand the examples on the internet as much as possible to help me understand how they work.

#!/usr/bin/python2.7

import random

class Perceptron:

    def __init__(self, weights=None, threshold=0.5):
        self.weights = weights
        self.threshold = threshold

    def activation(self, input_vector):
        # activation of the perceptron. 
        val = sum(value * self.weights[index] for index, value in enumerate(input_vector))
        if val > self.threshold:
            return 1
        else:
            return 0

    # learning rate is also the alpha
    def train(self, training_set, learning_rate = 0.1):
        # set the initial weights to random numbers in the range [-0.5,0.5] or 0
        self.weights = [0 for i in training_set[0][0]]

        while True:
            error_count = 0
            for input_vector, desired_output in training_set:
                print self.weights
                result = self.activation(input_vector)
                error = desired_output - result
                if error != 0:
                    error_count += 1
                    # weight training
                    for index, value in enumerate(input_vector):
                        #delta rule : learning rate * error * current value                    
                        self.weights[index] += (learning_rate * error * value)
            if error_count == 0:
                break

    def test(self, training_set):
        for input_vector, desired_result in training_set:
            if(self.activation(input_vector) != desired_result):
                return False
        return True

if __name__ == "__main__":

    # values for a AND gate.
    training_set_AND = [((0, 0), 0),
    ((0, 1), 0),
    ((1, 0), 0),
    ((-1, 1), 1)]

    training_set_OR = [((0,1), 1),
                       ((1,0), 1),
                       ((0,0), 0),
                       ((1,1), 1)]

    # values for a NAND gate.
    training_set_NAND = [((1, 0, 0), 1), 
                        ((1, 1, 0), 1), 
                        ((1, 0, 1), 1),
                        ((1, 1, 1), 0)
                        ]

    p = Perceptron(None, 0.5)
    p.train(training_set_AND, 0.1)
    print p.test(training_set_AND)

#!/usr/bin/python2.7

import random

class Perceptron:

def __init__(self, weights=None, threshold=0.5):

self.weights = weights

self.threshold = threshold

def activation(self, input_vector):

# activation of the perceptron.

val = sum(value * self.weights[index] for index, value in enumerate(input_vector))

if val > self.threshold:

return 1

else:

return 0

# learning rate is also the alpha

def train(self, training_set, learning_rate = 0.1):

# set the initial weights to random numbers in the range [-0.5,0.5] or 0

self.weights = [0 for i in training_set[0][0]]

while True:

error_count = 0

for input_vector, desired_output in training_set:

print self.weights

result = self.activation(input_vector)

error = desired_output - result

if error != 0:

error_count += 1

# weight training

for index, value in enumerate(input_vector):

#delta rule : learning rate * error * current value

self.weights[index] += (learning_rate * error * value)

if error_count == 0:

break

def test(self, training_set):

for input_vector, desired_result in training_set:

if(self.activation(input_vector) != desired_result):

return False

return True

if __name__ == "__main__":

# values for a AND gate.

training_set_AND = [((0, 0), 0),

((0, 1), 0),

((1, 0), 0),

((-1, 1), 1)]

training_set_OR = [((0,1), 1),

((1,0), 1),

((0,0), 0),

((1,1), 1)]

# values for a NAND gate.

training_set_NAND = [((1, 0, 0), 1),

((1, 1, 0), 1),

((1, 0, 1), 1),

((1, 1, 1), 0)

]

p = Perceptron(None, 0.5)

p.train(training_set_AND, 0.1)

print p.test(training_set_AND)

Lego Landrover

May 20, 2012 / Jack Concanon / 0 Comments

Just seen a fantastic video of a Landrover Defender 110 built entirely from lego; not just that but it has a full five speed sequential gear box, high and low range and a choice between two and four wheel drive (not entirely authentic that part)!

It’s really worth watching just to see the inner workings and fantastic little details such as moving pistons. Some more information can be found on Sheepo’s blog.

Also this seems like a pretty good excuse to post a picture of my full size version!

Pointless Error Messages

May 19, 2012 / Jack Concanon / 0 Comments

Gee, thanks Battlefield

So you’re sat there playing a game, about to pull off some spectacular moves and own some noobs when your game freezes, you fall through the floor and you are dumped back into the reality of the Windows Desktop. Quietly muttering some choice words about computers you discover a fantastic error message, informing you that ‘something went wrong’… Well, thanks error message, without your thoughtful take on everything I would have presumed to have teleported into a map that looks suspiciously like my desktop!

These error messages are becoming more and more common, and yes I fully understand that the alternative is the blue screen of death, which is equally unhelpful, but if you’re going to go to the effort of handling errors correctly without breaking everything why waste all that effort by just telling the user that something went wrong. How about, something went wrong and here is what it was?

Because I want a browser with sass

If it’s that important to try and hide the gory details of an error from the average user have a nice button called technical details. Average Joe isn’t going to click that, he still thinks an illegal operation involves the police.

How about some form of standard for displaying error messages? Maybe code red, yellow and green? Easy to understand for everyone.

Got any more examples of useless error messages?

Free Twitter Data Set

May 1, 2012 / Jack Concanon / 1 Comment

I needed a twitter data set for my thesis and struggled to find one that was freely available. I ended up downloading the data I needed so I thought I would release it here. It does come in a slightly annoying format which is a MongoDB dump file but it should be easy enough to extract to there and use it. The structure of each record is shown below.

So if you wanted to go over all records in the database you would do it like this –

from pymongo import Connection

if __name__ == "__main__":

    # connect to mongodb and get tweets collection
    db_connection = Connection()
    fyp_db = db_connection.fyp
    tweets_db = fyp_db.tweets
    num_tweets = 0

    for user_col in tweets_db.find():
        num_tweets += len(user_col['tweets'])
        for friend in user_col['friends']:
            if len(user_col['friends'][friend]) > 0:
                num_tweets += len(user_col['friends'][friend])

from pymongo import Connection

if __name__ == "__main__":

# connect to mongodb and get tweets collection

db_connection = Connection()

fyp_db = db_connection.fyp

tweets_db = fyp_db.tweets

num_tweets = 0

for user_col in tweets_db.find():

num_tweets += len(user_col['tweets'])

for friend in user_col['friends']:

if len(user_col['friends'][friend]) > 0:

num_tweets += len(user_col['friends'][friend])

Download Twitter Mongo Dump

Categories

Meta

MonthMay 2012

Language Detection Algorithm

Python Feed Forward Perceptron

Lego Landrover

Pointless Error Messages

Free Twitter Data Set

Recent Comments