Category Archives: Techie

Zip all changed files

Calling all server admins
Our server admin was off today and I had to set a site live. We have a dategrep.py script that lists all changed files since a certain date so that those files can then be ftp’d to the live server. This seemed far too tedious for me so I re-wrote the dategrep.py script so that given a date it will zip up all changed files since that date. The zip archive will have the same directory structire so that all you have to do is extract that archive on the live server and you’re done.
[code lang=”python”]
“””
Checks for files modifies after a given date time and then adds them to a zip
file that can be easily extracted
“””

import os, time, sys
import zipfile
import glob
from datetime import datetime

class SimpleZipDateGrep:
today = time.time()
yesterdayTimestamp = today-24*60*60
yesterday = datetime.fromtimestamp(yesterdayTimestamp)
iYear = yesterday.year
iMonth = yesterday.month
iDay = yesterday.day
iHour = yesterday.hour
iHourOrig = iHour
iMin = yesterday.minute
zFileName = “SimpleZipDateGrepChange.log.zip”
logFileName = “SimpleZipDateGrepChange.log”
sExcludeFiles =””
bExcludeFiles = False
lExcludeFiles = [‘list.txt’,’log.txt’]

sYear,sMonth,sDay,sHour,sMin = “”,””,””,””,””

def __init__(self):
self.GetArgs()
self.ZipFiles()
print “Finished”

def GetArgs(self):
self.sYear=str(raw_input(“Year (yyyy)[default: “+str(self.iYear)+”]: “))
self.sYear = self.sYear.strip()
if self.sYear != “”:
try:
self.iYear = int(self.sYear)
except:
print “This is not a valid year”
#exception in casting to int

self.sMonth=str(raw_input(“Month (mm)[default: “+str(self.iMonth)+”]: “))
self.sMonth = self.sMonth.strip()
if self.sMonth != “”:
try:
self.iMonth = int(self.sMonth)
except:
print “This is not a valid month”
#exception in casting to int

self.sDay=str(raw_input(“Day (dd)[default: “+str(self.iDay)+”]: “))
self.sDay = self.sDay.strip()
if self.sDay != “”:
try:
self.iDay = int(self.sDay)
except:
print “This is not a valid day”
#exception in casting to int

self.sHour=str(raw_input(“Hour (hh)[default: “+str(self.iHour)+”]: “))
self.sHour = self.sHour.strip()
if self.sHour != “”:
try:
if int(self.sHour) < 25: self.iHour = int(self.sHour) except: print "This is not a valid hour" + str(self.sHour) #exception in casting to int

self.sMin=str(raw_input(“Minute (mm)[default: “+str(self.iMin)+”]: “))
self.sMin = self.sMin.strip()
if self.sMin != “”:
try:
if int(self.sMin) < 61: self.iMin = int(self.sMin) except: print "This is not a valid minute" + str(self.sMin) #exception in casting to int

self.sExcludeFiles = str(raw_input(“Exclude Files (y|n)[default: n]: “))
if self.sExcludeFiles.lower() == ‘y’:
self.bExcludeFiles = True
else:
self.bExcludeFiles = False

def ZipFiles(self):
”’
0 tm_year (for example, 1993)
1 tm_mon range [1,12]
2 tm_mday range [1,31]
3 tm_hour range [0,23]
4 tm_min range [0,59]
5 tm_sec range [0,61]; see (1) in strftime() description
6 tm_wday range [0,6], Monday is 0
7 tm_yday range [1,366]
8 tm_isdst 0, 1 or -1; see below
”’
iBaseTime=time.mktime((self.iYear,self.iMonth,self.iDay,self.iHour,self.iMin,0,0,0,0))

if self.iHour != self.iHourOrig:
iBaseTime = iBaseTime + time.altzone

currentFileName = os.path.basename(str(sys.argv[0]))
f=open(self.logFileName,’w’)
zFile = zipfile.ZipFile(self.zFileName, “w”)

f.write(“- File changed since – %s\n”%time.asctime(time.localtime(iBaseTime)))

#get a list of all the root level directories
curDir = os.curdir

for root,dirs,files in os.walk(curDir):
if root.lower().find(“svn”)>-1 or root.lower().find(“cvs”)>-1 :
continue
for name in files:
if name.lower().find(“svn”) > -1 or name.lower().find(“cvs”)>-1 or name == self.zFileName or name == self.logFileName or name == currentFileName:
continue
iTime=os.path.getmtime(os.path.join(root,name))
if iTime and iTime > iBaseTime:
#add the file to the zip file
if self.bExcludeFiles == True or (self.bExcludeFiles == False and root not in self.lExcludeFiles and name not in self.lExcludeFiles):
#Products
curPath = os.path.join(root,name)
curPath = curPath.replace(“.\\”,””)
if curPath.find(‘.’)==0:
curPath = curPath[1:]
print curPath
f.write(“%s %s\n”%(time.asctime(time.localtime(iTime)),curPath))
zFile.write(curPath, curPath, zipfile.ZIP_DEFLATED)
zFile.close()
f.flush()
f.close()

if __name__ == ‘__main__’:
SimpleZipDateGrep = SimpleZipDateGrep()

[/code]

Download SimpleZipDateGrep.zip {zip 1Kb}. It’s also on the downloads section. 🙂

Update:
Some bug fixes and now written in an OO stylie for reusablility

ASPX craziness

There I was hammering away, well in the zone writing a pretty beefy login system in C# when all of a sudden all the pages stopped working. I was running the application on my own machine so I tried to debug by setting a breakpoint and attaching the aspnet worker process. Nope just output blank pages. I was about to go back to a previous version of the project I had in CVS but I had done a good bit of work and didn’t fancy trying to merge it all back in bit by bit. I then noticed the following on another aspx.cs file that I had not been working on .

[code lang=”cpp”]
#region Web Form Designer generated code
override protected void OnInit(EventArgs e)
{
//
// CODEGEN: This call is required by the ASP.NET Web Form Designer.
//
InitializeComponent();
base.OnInit(e);
}

///

/// Required method for Designer support – do not modify
/// the contents of this method with the code editor.
///

private void InitializeComponent()
{
this.Load += new System.EventHandler(this.Page_Load);

}
#endregion
[/code]

I had seen this in my login.aspx.cs but stripped it out as thought it was the usual Visual studio bullshit. I added it and hey presto, I have XHTML being output and I can debug to my hearts content.

The moral of todays little tale – Don’t ever delete something if you don’t know what it does.

That’s enough for today, I’m going home (in my Volvo 🙂 )

XSLT output CDATA sections

I was transforming some xml to xml another format and some of the text nodes could potentially contained invalid characters, so placing them in CDATA sections would be wise. I tried the following –

[code lang=”xml”]

…….

< ![CDATA[
]]>

[/code]

which of course did not work and resulted in what you see below. I knew this would happen but I was real busy and figured it worth a shot.

[code lang=”xml”]

<xsl:value-of select=”fields/field[normalize-space(@name)=’Description’]/value/text()” />

[/code]

Some digging resulted in the finding of the “cdata-section-elements” attribute which contains a whitespace-separated list of QNames so using –

[code lang=”xml”]

…….



[/code]

I get the desired result

[code lang=”xml”]
< ![CDATA[This is a description]]>
[/code]

Marvellous I tell you, It’s amazing what you can actually learn when you RTFM , in this case the XSL W3C recommendation.

ValueError: bad marshal data

Very strange error happened today – I was running a few python scripts I was writing when I suddenly started getting a “ValueError: bad marshal data” error .

ValueError: bad marshal data

It was due to libxml2 but I didn’t know why. I deleted the “C:\Python24\Lib\site-packages\libxml2.pyc” and the “C:\Python24\Lib\site-packages\libxml2.pyo” file which solved the problem. No idea why as the error message means nothing to me.

Python Publisher Posted Vars

I was trying to get all of the variables posted to a python script added to a dictionary and was trying
[code lang=”python”]
form = cgi.FieldStorage()
[/code]
but it seems that you cannot use FieldStorage when using the publisher handler in mod_python. As the request variable req is always available, use the following.
[code lang=”python”]
styleArgs = {}
for k in req.form.keys():
styleArgs[k] = “‘”+req.form[k]+”‘”
[/code]

I couldn’t find any docs on this so I thought I’d post it.

What’s my bloody session ID

While having a wander over at xulplanet.com I stumbled across the Cookie Manager Interface (Interface to the Gecko engine that is).

We have a simple HTML page in work that we use to emulate Payment Service Providers[PSP] (like Paypal or SecureTrading). It’s a right pain to test as we have to edit the page each time we test a different site. We also have to find our session ID from that sites cookie if we want to test it.

I wrote a few functions to allow me to populate a select drop down with the cookie domains (and session IDs) for a filtered list of all my cookies. On Page load the drop down is genereated and onChange of that dropdown the forms are updated with the chosen sites SessionID and the forms actions is also changed. Pretty neat and saves me about 10 clicks everytime I need to test my payment emails or that the PSP integration is working.

[code lang=”javascript”]
function _loadCookies()
{

//need to ask for permission to look at the users cookies
try {
netscape.security.PrivilegeManager.enablePrivilege(“UniversalXPConnect”);
} catch (e) {
alert(“Permission to save file was denied.”);
}
var nsICookie = Components.interfaces.nsICookie;
var _cm = Components.classes[“@mozilla.org/cookiemanager;1”]
.getService(Components.interfaces.nsICookieManager);

var _ds = Components.classes[“@mozilla.org/intl/scriptabledateformat;1″]
.getService(Components.interfaces.nsIScriptableDateFormat);
var _hosts = {};
var e = _cm.enumerator;
var hostCount = { value: 0 };

while (e.hasMoreElements()) {
var cookie = e.getNext();
if (cookie && cookie instanceof Components.interfaces.nsICookie)
{

var strippedHost = _makeStrippedHost(cookie.host);
if (!(strippedHost in _hosts) || !_hosts[strippedHost])
{
_hosts[strippedHost] = { cookies : [],
rawHost : strippedHost,
level : 0,
open : false,
container : true };

++hostCount.value;
var c = _makeCookieObject(strippedHost, cookie);
_hosts[strippedHost].cookies.push(c);
}
}
else
break;
}

var _select = document.createElement(‘SELECT’)
for (var host in _hosts) {
var cookies = _hosts[host].cookies;
var hostName =_hosts[host].rawHost;
var rExp = /kf/gi;
var rExpResults = hostName.match(rExp);
if(rExpResults && rExpResults.length > 0)
{
for (var _cookie in cookies)
{
var _option = document.createElement(‘OPTION’)
_option.innerHTML = _hosts[host].rawHost;
_option.value = cookies[_cookie].value;
_select.appendChild(_option);
}

}
}
_select.id=”siteURLChange”;
_select.onchange = changeActionAndSessID;

var URLForm = document.getElementById(‘siteurl’);
var URLInput = document.getElementById(‘url_site’);
URLInput.style.display = “none”;
insertAfter(URLForm, _select, URLInput);
changeActionAndSessID();
}

function _makeStrippedHost(aHost)
{
var formattedHost = aHost.charAt(0) == “.” ? aHost.substring(1, aHost.length) : aHost;
return formattedHost.substring(0, 4) == “www.” ? formattedHost
.substring(4, formattedHost.length) : formattedHost;
}

function _makeCookieObject(aStrippedHost, aCookie)
{
var host = aCookie.host;
var formattedHost = host.charAt(0) == “.” ? host.substring(1, host.length) : host;
var c = { name : aCookie.name,
value : aCookie.value,
isDomain : aCookie.isDomain,
host : aCookie.host,
rawHost : aStrippedHost,
path : aCookie.path,
isSecure : aCookie.isSecure,
expires : aCookie.expires,
level : 1,
container : false };
return c;
}
[/code]

I hope this code helps someone wanting to view cookies from a certain domain or from a filtered list.

Mono XSP 404

Arghh!!!

I worked form home yesterday as I had the plumber round power flushing the central heating system. I spent most of the day trying to get mono xsp working (on Ubuntu on my ibook) which I did and all the sample .net applications were working great.

Mono XSP 404 ErrorI came in to work today to show my boss what I had discovered and how it could be of use to us… and what do you know mon-xsp stopped working. I cannot fathom what has changed. All I did since it worked was reboot.

I have had a look around but all I found out was that the errors related to compilation errors. I can’t see how this happened as they were compiled yesterday and worked fine, are they recompiled each time??

I have reinstalled mono-xsp2-base, mono-apache-server2, mono-xsp , asp.net examples. Nada same error again The server is run using

/etc/init.d/mono-xsp start

Any ideas? Also anybody know how to get mod_mono installed on Linux PPC without having to recompile apache? and what the hell is apr-config?

Update
After some more playing around I re-installed mono-mcs package. I had more recent package over the ubuntu default which messed things up seemingly. Reverting to the ubuntu version solved and it and I am now serving up .Net applications on Linux on Mac 🙂

What a pillock

Damn Dan Brown and his “The DaVinci code“. This morning I started to get an earlier train as they have changed the timetable, meaning it is dark when I get to Abergele. With it being dark and the fact that I was so engrossed in my book, I missed my stop. I jumped out of my chair “Bollocks” just as we pulled away. I managed to get a train back from the next station but man do I feel a pillock.