The Python Subprocess module is a highly flexible way of running background processes in Python. Here are two functions that show how to run external shell commands, one will wait for the command to return and the other runs in the background allowing the controlling script to stop it as required.

Passing unsanitized input to these functions will not end well.


Generic CSV to Plot With Numpy and Python

Given multiple CSV files containing a timestamp column and some plottable values (int or float) this script will plot multiple columns of data by time.

Usage is as such – ./CSVPlot.py first.csv second.csv third.csv



Time Bound Python Queue

An extension of the standard Python queue that only stores elements for a given number of seconds before being removed. Useful if you don’t know the volume of data being added to the queue but need to limit it in some way.

There is no decrease in performance when removing items.

Without item removal 

Queue size: 10000

real 0m10.771s
user 0m0.217s
sys 0m0.028s

With item removal 

Queue size: 4659

real 0m10.765s
user 0m0.203s
sys 0m0.028s


SQLite Database to CSV

Wrote a Python script that can dump a SQLite database to CSV, it takes the column names as the header and then writes values below. It also has a –sumarise option that can create a one line CSV file of multiple lines in a database.

Does have the option to support XML output in the future but a format would need to be defined by the user.


Python Socketed Receiver

This is a quite cool little class I wrote for receiving log messages over sockets. It is launched as a thread and uses a python Queue to store messages for retrieval by consumers in your main program. It also means I can have multiple sockets for different log sources.


Correct Setup of Python Easy-Install

Starting work on some python driver development today for the ant+ system (more on this later) and had to install a bunch of Python modules that didn’t appear on the standard Ubuntu repos.

Python has a very easy package manager of it’s own called easy tools which is found in the python-setuptools package. But this isn’t everything you need, you’ll receive a lot of errors when trying to use easy tools such as –

msgpack/_msgpack.c:4:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command ‘gcc’ failed with exit status 1

In order to resolve these errors you’ll need to ensure you have g++ installed as well as the python header and static library files. These two packages are called g++ and python2.7-dev. Change the python version depending on your preference or download the python-all-dev package.

Language Detection Algorithm

My thesis required dealing with large quantities of Twitter data and to make it easier for myself I decided to only use English language tweets. Due to the ‘interesting’ grammar and spelling that is frequently found on line a different algorithm to a standard dictionary test was required. This one checks for common English n-grams (letter groupings) and returns a value of how sure the input is English.

Here’s some examples:

Good morning everyone! : 66%
Im So Sick. Really in a bad Position : 50%
Thats the fear of unicorns : 100%
Gnt too com uma saudade daminha namorada vcs nao tem ideia :(((( : 0%
Ils testent les hologrammes pour faire des concerts par des morts. Bientôt même les pas morts feront des hologrammes et vous irez les voir. : 12%
il7ain ilnass ykhl9un men habat twitter o yntqlun to istagram=))!!! : 40%


Python Feed Forward Perceptron

I thought I would post a couple of useful bits of code from my final year of university.

The first is a feed forward perceptron written in Python. I tried to expand the examples on the internet as much as possible to help me understand how they work.

Free Twitter Data Set

I needed a twitter data set for my thesis and struggled to find one that was freely available. I ended up downloading the data I needed so I thought I would release it here.  It does come in a slightly annoying format which is a MongoDB dump file but it should be easy enough to extract to there and use it. The structure of each record is shown below.

So if you wanted to go over all records in the database you would do it like this –

Download Twitter Mongo Dump

Code Golf Progress Bar

Had a go at some code golf today, went for this challenge: http://codegolf.stackexchange.com/questions/5382/create-an-ascii-progress-bar

Here is my solution which is 125 characters small.

So the output for this would be as such:

Pi Approximator in Python Using Liebniz Formula

Calculates Pi to an increasing degree of accuracy using Leibniz formula. The more cycles that it runs for then the greater the accuracy of the result.

Python Watchman System Monitor

A ‘live’ console based systems monitor in Python. Can be easily extended to monitor various other things. Currently only monitors disk usage, memory usage and zombie processes. Can provide advice for a user. Written to aid some non technical system users. Update: Added percentage bars and tidied up. Also added a CPU monitor.

© 2022 Acodemics

Theme by Anders NorénUp ↑