Category Archives: python

Archive of an SVN changeset

All too often I need to release changes to a staging or live server and I always felt the releases were taking too long.

At Karova we use SVN and I always try and commit any releases in one changeset. I wrote a python script to archive all the files listed in a changeset. It’s been very very useful so far. Simply run the python script in the root of your repos and you will be prompted for the changeset number. The script then archives all the files in that changeset in a zip file named by changeset. You’ll need the svn client in your path for it to work though.

svnarchivebychangeset.zip

Enjoy.

mp3 id3 tag whatsit

I bought a few mp3s from mp3sparks (formerly allofmp3) and was annoyed that the mp3s were not tagged with their track numbers but they were in the filename.

I had a quick look to see if I could fix this somehow. I found id3-py which I used to write a python script that fixes this and the artist,title and comment. One thing I noted was that you can only have a max of thirty characters in your track title. With a “Various Artists” album you want the artist in the title (Album artist is not in the id3 spec but it is supported by itunes and WMP). The concatenation of title and artist is quite often more than 30 characters – not good. Anyone found a way to write “Album Artist” to an mp3 using python?
mp3sparks is great but why in all things upside-down-monkey-drunk do I have to download each file individually and why do I have to right click (it is possible force a download guys).

I’ll upload an exe for installation that fixes your mp3sparks mp3s later.

Halfviking get’s videos (ffmpeg, convert, flvmdi, flvplayer)

I’ve always wanted my online gallery export script to handle videos and last weekend, I bit the bullet and wrote it. I used quite a few Open source projects to accomplish this which means that PEAExport has a few dependencies but I think it’s worth it. My Canon Ixus takes AVI movies. The sizes of the movies range from 1MB to 50MB. To upload these would just be silly and each viewer would have to have an AVI viewer, which probably wouldn’t load in the browser. I really like flash movies players like those found on YouTube and Google Video so I anted to find a free one to use, I found the wonderful flvplayer @ http://www.jeroenwijering.com/. It’s an excellent player and free for non-commercial use.

So at this stage, I have my Videos in AVI format and a flash video (flv) player. I need to transcode them from AVI to FLV to work with the player. In work a few weeks ago we came across the SUPER application. This is a GUI front end to FFMPEG which is an absolutely wonderful utility, It handles tens of formats and hundreds of options. I used the FFMPEG.exe bundled with SUPER to tanscode them from AVI to FLV

ffmpeg.exe -i 'video.avi' -y -ab 64 -ar 22050 -b 200 -r 25 -s 650*450 'video.flv'

The options above (full list of options):

  • -i ‘video.avi’= input file is video.avi
  • -y = overwrite existing file if it exists
  • -ab 64 = audio bitrate is 64 kbit/s
  • -ar 22050 = audio sampling rate is 22050 Hz
  • -b 200 = video bitrate is 200 kbit/s
  • -r 25 = frame rate is 25Hz
  • -s 650*450 = rame size is 650 pixels by 450 pixels (WxH)
  • ‘video.flv’ = output file

Running this converts the hefty AVI video to a lightweight (relatively) FLV video. I then need to get the flvplayer playing this newly generated FLV.

<object data=”flvplayer.swf” type=”application/x-shockwave-flash”>
<param value=”file=video.flv&image=medium.jpg&linkfromdisplay=true&link=../14″ name=”flashvars” />
<param name=”movie” value=”flvplayer.swf?file=video.flv&image=medium.jpg&linkfromdisplay=true&link=../14″ />
</object>

There are many options for the flvplayer (full list of options). I only needed a few –

  • file=video.flv = input file is video.flv
  • image=medium.jpg = the loading image before you press play is medium.jpg
  • linkfromdisplay=true = when the user clicks directly on the movie area then they will be redirected to another page
  • link=../14 = when they click on the video, they will be redirected to the relative link “../14”

This plays fine but the progressbar isn’t running, This is a known limitation of FFMPEG (it’s fixed in the SVN version), FFMPEG does not write the metadata like video length etc to the FLV file. To do this, I was able to use another free utility called FLVMDI

flvdmi.exe video.flv video.flv

Running this inserts all the required metadata into the FLV video. The progress Bar now works perfectly.

You’ll notice that in the options for the flvplayer, I included “image=medium.jpg”. I could have used any image here but I wanted to use a frame from the video. Fortunately FFMPEG can do this.

ffmpeg.exe -i 'video.avi' -y -vcodec png -vframes 1 -an -f rawvideo -s 650x450 'videoimage.png'

The options above (full list of options):

  • -i ‘video.avi’= input file is video.avi
  • -y = overwrite existing file if it exists
  • -vcodec png = force video codec to png
  • -vframes 1 = set the number of video frames to one
  • -an = disable audio
  • -f rawvideo = force format to rawvideo
  • -s 650*450 = frame size is 650 pixels by 450 pixels (WxH)
  • ‘videoimage.png’ = output file

This results in a PNG file, quite a big one, taken from the first frame of the video. The file was too big for web use so I needed to convert it to JPEG. I could have used the Python image library PIL to do this (PEAExport already uses PIL) but I was already using some command utilities utilites to create videos and images. I thought, there’s no harm in another. That other was Convert.exe from the ImageMagick Library.

convert.exe -quality 75 videoimage.png videoimage.jpg

The options above (full list of options):

  • -quality 75 = compression level set to 75%
  • videoimage.png = videoimage.png is the input file
  • videoimage.jpg = videoimage.jpg is the output file

Excellent, I now have a lightweight flv video but I want a thumbnail to display in my gallery index pages. I ran the FFMPEG fram grap command but with “-s 250×186” to reduce the size to thumbnail size. I also want to indicate that it’s a video that the users is going to see and not another photo. So I need to Annotate it with the word “Video”

convert.exe -quality 75 -gravity South -font C:WINDOWSFontsGOTHICB.TTF -pointsize 50 -fill black -annotate +1+1 Video -fill white -annotate +0+0 Video videoimage.png videoimage.jpg

The options above (full list of options):

  • -quality 75 = compression level set to 75%
  • -gravity South = direction primitive gravitates to when annotating the image – In this case, we want the annotation loacted in the South of the image
  • -font C:\\WINDOWS\\Fonts\\GOTHICB.TTF = The font for the annotation. In this case I am using Century Gothic Bold
  • -pointsize 50 = The font size is set to 50 points
  • -fill black = The fill of the annotation is to be black
  • -annotate +1+1 Video = Annotate the word “Video” plus one pixel in both directions from the base position of “South”
  • -fill white = The fill of the next annotation is to be white
  • -annotate +0+0 Video = Annotate the word “Video” plus zero pixels in both directions from the base position of “South”
  • videoimage.png = videoimage.png is the input file
  • videoimage.jpg = videoimage.jpg is the output file

This creates a thumbnail image with the word “Video” in white with a black drop shadow, in Century Gothic Bold font, size 50 points.

thumbnail image with the word Video

And that my friends is it. Enjoy all my vidoes @ http://www.halfviking.com. I have also created a new page @ http://www.halfviking.com/videos using flvplayer’s playlist feature. My favourite single video is Bobby very bemused as to what on earth is planted in the ground at Crosby beach north of Liverpool.

Christmas Charity – the Open Source way

Every year for the last few years I have given to charity at Christmas. Last year it was Oxfam but this year I wanted to do something a bit different. In the early days of Aggreg8 development, I got a $10 donation and it freaked me out, I coded for days with a crazy energy and a boosted ego, knowing that someone thought that what I was doing was worth some of their hard earned cash. So this year, I have decided to give $5 to 10 different Open Source projects. The projects I have chosen are ones that I use on a day to day basis and really admire.

If we could all do something like this then Open source would continue to thrive and we will all reap the benefits for years and years to come. Joel noted that most people don’t want credit for donating but I do as donating is clearly not a selfless act ;).

auto rewrite rules and google sitemap generation

Although we’re very busy in work at the moment, the last few evenings I have been pythoning myself up and have come up with 3 nifty little scripts.

Usage of all three is quite simple –

python scriptName.py http://www.karova.com

There is also a zip archive for download. As you can see I have started using Google Code project hosting. It suits my needs perfectly as it’s simple to use and the SVN server they are using is hellish quick. I’ve named the project acr (Auto Create RewriteRules). For a sample of output see http://www.karova.com/sitemap.xml . The rewrite rules was written as a personal project but also for KarovaStore – So expect release 2.1 to have clean urls and auto google sitemaps.

Update:

Yes I know Google have their own sitemap generator but mine is alot easier to use.

Dev crazy

In an attempt to vary the type of programming I am doing, out of office I have been working on quite a few cool projects. Outlined a few below – unfortunately none are for download yet as they are still in progress.

KarovaDev Firefox extension
As Karova‘s main product is a hosted e-commerce solution, we naturally have to integrate with quite a few payment service providers. It can be a chore to test our stores with these payment service providers, having to enter dummy or real credit card details everytime. I hate monkey work so I wrote a firefox extension for the Karova developers to help in the testing of Karova stores. It’s pretty sweet, You can edit all the default data passed for each payment service providers, specify the domains or virtual directories (if installed on localhost) to test against. The final test page scans all the user’s cookies and chooses the ones that match the supplied domains etc. You can then choose your site and test to your heart’s content- saving hours of monkey work.
Screenshot of KarovaDev firefox extension
It was nice to work on a beefy extension again. It re-affirmed to me how good the Gecko/ XUL platform is.
Philroche Blog Tool
Most of the C# work I do in wrk is web applications running on IIS. I do some command line apps that run as scheduled tasks but I don’t have cause to develop any GUI apps. Microsoft released Visual Studio 2005 express so I downloaded it and gave WinForms a shot… easy peasy to get a nice GUI designed and built. I decided to write a simple RSS aggregator to see how easy it would be.
Philroche Blog tool view feed
I haven’t written the RSS parser yet but I have the tree view and the Mozilla ActiveX control working well. I could have used the built in webbrowser (IE) component but then why would I want to do that.
Philroche Blog tool make post
I wanted to see if I could integrate with the wordpress API too to make blog posts so I turned to Xstandard (the lite version). Xstandard also have a .NET component that you can use. It is really neat and you can bet your house on the markup being valid.

Visual studio 2005 express is excellent and free too (as in beer). Also checkout SQL server 2005 express which I have used on a very beefy project already and it is ace – also free (as in beer).

Mod_Python
Since having a fairly new Dell machine donated to me by my Dad I have been feverishly using Ubuntu. I have Apache and mod_python running and have a nice little app running. Mod_python as a development platform is sweet as it is not as contstrained as TurboGears or Ruby on Rails. It is very hard to debug though.

So all in all I am getting some nice variety in my dev work, I find it can sometimes get stale if you’re doing the same thing day-in day-out, especially if it involves legacy asp sites. Is there anything you guys would recommend trying out as an antidote to work work while still being dev work (if you know what I mean). I do want to look at mono and PyGTK.

Archive date grep V1.0

Archive date grep is a python script that allows you to archive (choosing zip or tar) all files within a directory structure which have been modifies since a user defined date. The archive maintains the original directory structure which makes it ideal for releasing websites. You can downlaod the script as a zip archive[zip 2 KB] or as a tar archive [tar 8 KB]. This script is alot better than the original SimpleZipDateGrep that I had on the site. Enjoy 🙂

Zip all changed files

Calling all server admins
Our server admin was off today and I had to set a site live. We have a dategrep.py script that lists all changed files since a certain date so that those files can then be ftp’d to the live server. This seemed far too tedious for me so I re-wrote the dategrep.py script so that given a date it will zip up all changed files since that date. The zip archive will have the same directory structire so that all you have to do is extract that archive on the live server and you’re done.
[code lang=”python”]
“””
Checks for files modifies after a given date time and then adds them to a zip
file that can be easily extracted
“””

import os, time, sys
import zipfile
import glob
from datetime import datetime

class SimpleZipDateGrep:
today = time.time()
yesterdayTimestamp = today-24*60*60
yesterday = datetime.fromtimestamp(yesterdayTimestamp)
iYear = yesterday.year
iMonth = yesterday.month
iDay = yesterday.day
iHour = yesterday.hour
iHourOrig = iHour
iMin = yesterday.minute
zFileName = “SimpleZipDateGrepChange.log.zip”
logFileName = “SimpleZipDateGrepChange.log”
sExcludeFiles =””
bExcludeFiles = False
lExcludeFiles = [‘list.txt’,’log.txt’]

sYear,sMonth,sDay,sHour,sMin = “”,””,””,””,””

def __init__(self):
self.GetArgs()
self.ZipFiles()
print “Finished”

def GetArgs(self):
self.sYear=str(raw_input(“Year (yyyy)[default: “+str(self.iYear)+”]: “))
self.sYear = self.sYear.strip()
if self.sYear != “”:
try:
self.iYear = int(self.sYear)
except:
print “This is not a valid year”
#exception in casting to int

self.sMonth=str(raw_input(“Month (mm)[default: “+str(self.iMonth)+”]: “))
self.sMonth = self.sMonth.strip()
if self.sMonth != “”:
try:
self.iMonth = int(self.sMonth)
except:
print “This is not a valid month”
#exception in casting to int

self.sDay=str(raw_input(“Day (dd)[default: “+str(self.iDay)+”]: “))
self.sDay = self.sDay.strip()
if self.sDay != “”:
try:
self.iDay = int(self.sDay)
except:
print “This is not a valid day”
#exception in casting to int

self.sHour=str(raw_input(“Hour (hh)[default: “+str(self.iHour)+”]: “))
self.sHour = self.sHour.strip()
if self.sHour != “”:
try:
if int(self.sHour) < 25: self.iHour = int(self.sHour) except: print "This is not a valid hour" + str(self.sHour) #exception in casting to int

self.sMin=str(raw_input(“Minute (mm)[default: “+str(self.iMin)+”]: “))
self.sMin = self.sMin.strip()
if self.sMin != “”:
try:
if int(self.sMin) < 61: self.iMin = int(self.sMin) except: print "This is not a valid minute" + str(self.sMin) #exception in casting to int

self.sExcludeFiles = str(raw_input(“Exclude Files (y|n)[default: n]: “))
if self.sExcludeFiles.lower() == ‘y’:
self.bExcludeFiles = True
else:
self.bExcludeFiles = False

def ZipFiles(self):
”’
0 tm_year (for example, 1993)
1 tm_mon range [1,12]
2 tm_mday range [1,31]
3 tm_hour range [0,23]
4 tm_min range [0,59]
5 tm_sec range [0,61]; see (1) in strftime() description
6 tm_wday range [0,6], Monday is 0
7 tm_yday range [1,366]
8 tm_isdst 0, 1 or -1; see below
”’
iBaseTime=time.mktime((self.iYear,self.iMonth,self.iDay,self.iHour,self.iMin,0,0,0,0))

if self.iHour != self.iHourOrig:
iBaseTime = iBaseTime + time.altzone

currentFileName = os.path.basename(str(sys.argv[0]))
f=open(self.logFileName,’w’)
zFile = zipfile.ZipFile(self.zFileName, “w”)

f.write(“- File changed since – %s\n”%time.asctime(time.localtime(iBaseTime)))

#get a list of all the root level directories
curDir = os.curdir

for root,dirs,files in os.walk(curDir):
if root.lower().find(“svn”)>-1 or root.lower().find(“cvs”)>-1 :
continue
for name in files:
if name.lower().find(“svn”) > -1 or name.lower().find(“cvs”)>-1 or name == self.zFileName or name == self.logFileName or name == currentFileName:
continue
iTime=os.path.getmtime(os.path.join(root,name))
if iTime and iTime > iBaseTime:
#add the file to the zip file
if self.bExcludeFiles == True or (self.bExcludeFiles == False and root not in self.lExcludeFiles and name not in self.lExcludeFiles):
#Products
curPath = os.path.join(root,name)
curPath = curPath.replace(“.\\”,””)
if curPath.find(‘.’)==0:
curPath = curPath[1:]
print curPath
f.write(“%s %s\n”%(time.asctime(time.localtime(iTime)),curPath))
zFile.write(curPath, curPath, zipfile.ZIP_DEFLATED)
zFile.close()
f.flush()
f.close()

if __name__ == ‘__main__’:
SimpleZipDateGrep = SimpleZipDateGrep()

[/code]

Download SimpleZipDateGrep.zip {zip 1Kb}. It’s also on the downloads section. 🙂

Update:
Some bug fixes and now written in an OO stylie for reusablility

Half Viking Gallery Goes Live

HalfViking.com
It’s been a while on the back burner but this weekend I went hell for leather and managed to get the necessary coding done to get Halfviking.com live.

I manage all my photos 6000+ photos in Photoshop Elements and I am very happy with how it copes but it’s web gallery export sucks ass.

What I found was that PSE stores it’s data in an Access database. I wrote (using parts and few sub routines form album2gallery) a perl script to extract the data in the DB and write it to XML.

I then wrote a wad of python to parse the XML and create a static version of the gallery you now see. Using the Python Imaging Library I also managed to resize all the images to thumbnails and larger versions. All my photos are up now…. I have nothing to hide 🙂 .

There is still a couple of hitches (some duplicate albums [now fixed :)] and it’s not browser tested and I want to add some funky javascript) but I couldn’t be arsed waiting any longer… enjoy (or not)