Random Musings of an Insane Mind

One of the biggest problems I had with the old control panel was that certain functions really weren’t well thought out, so, using the control panel was quite cumbersome. Some of the earlier redesigns of the Task Status Report made a large impact. Making the report easy to read and to convey the critical information very quickly was one of the goals.

The primary goal was to make sure that it was as easy to use the Control Panel to do things as it was to start an ssh session and do things manually. But, we needed to make the form easy enough for people to use that didn’t know what they needed.

Easy enough for everyone, still quick enough for someone that knows precisely what they want.

The following two screenshots are from the old Control Panel. Arguably one could say that the Simple interface was harder to use than the Advanced interface.

Simple Cronjob Entry Form

Advanced Cronjob Entry Form

The replacement Cron Job entry page:

With a few buttons to allow very commonly entered options to be selected, adding a cron job becomes quite easy. I have debated whether to add a single line entry that just takes the raw crontab entry so that people don’t need to parse it and fill in the form.

Sometimes, forms can be usable.

Earlier today I was debating Ajax Push and Pyramid for a project I had in mind. I ended up spending about 45 minutes writing a quick proof of concept, then, decided that perhaps something a bit more detailed with some documentation would be helpful for others.

I used Pyramid and APE and wrote a quick demo app. All of the code for the demo app can be downloaded from http://code.google.com/p/pyramid-ape-demo/.

In the html/ directory, the files, graphics and javascript files required to run the client side of the app are included. In the ape_server/ directory, the javascript that needs to be installed in the Ape Server scripts/ directory is present. You’ll want to modify the password. Also included in the html/ directory is a python script called push.py which allows you to use urllib2.urlopen to communicate with the server directly. And finally, in the ape/ directory is a very minimal Pyramid application. pyramidape.wsgi is also included as a starting point to get the site set up.

In the demo, the left hand Coke can is controlled completely by the Ape Javascript Client code. Communications between the browser and Ape server are not processed by anything but Ape. On the right hand side, the Coke can is controlled by a json post to Pyramid and then Pyramid uses urllib2.urlopen to communicate with Ape which then updates the page.

Changes made on the page are reflected among all of the other people that are currently viewing the page in realtime. Since we’re using Ajax push, the page doesn’t need to be reloaded to show those changes. In this example, an img src and the alt text is changed along with a button. You can write your script to modify any html on the page – changing the colors of the page, elements, etc.

Using Ajax push and long polling with Pyramid isn’t difficult and this simple demo and example code should be a good starting point.

I ran into an issue with a library that did email validation. One of the client’s email addresses didn’t validate and while debugging the code, I realized that more than just that client’s email address could be rejected. While using a regexp is not a failsafe solution to email address validation, it does attempt to make sure you are getting a reasonably good email address.

However, the regexp can still fail and in some cases, you get a client that is turned away. The only true way to validate an email address is valid at least once is to require a client to receive an email and validate their account. With so many throwaway account services, you can never be assured that anything past that first email address will ever be received. However, to give you the best chance at getting a valid email address, you need to do some validation.

There are a number of methods for validating email addresses on the internet and most fail somewhere. Do you accept emails with ” or < > in them? You might, since some email clients paste that information when you copy an email address. Or, do you accept the bare minimum email address, accepting the fact that people could use -, +, . or even ‘ in the name portion? Trying to get the person to enter a name that is reasonably valid is the goal.

One could have a very generic rule that looks for an @ with a . after it and some characters. That would ensure almost the largest cross-section of potential email addresses but we may have cast too wide a net. We could go with a very strict regexp but potentially miss out on some corner cases.

As we can see, the error message presented here is correct, the email address we’ve entered is not correct:

But, what if we do put in a valid email address:

Our regexp fails to recognize the email address even though it is valid. Of course, this requires someone to adjust some code and we’re fine until the next email address that fails our test is entered.

What if we introduce a new error condition, a soft fail. If the address passes our very simple test and looks to have a valid email address construction, we’ll put another field asking the user to check to see if any typos were made. Users that pass our more stringent regexp wouldn’t be subjected to the soft fail error handling.

When a user is subjected to the soft fail, our software can log the email address so we can adjust our validation rules, but, we’re not restricting a client from being able to sign up because their email address didn’t fit the regexp that we’re using.

While I feel reasonably confident with some of the regexp’s I use, email validation has always been one of the most troublesome portions of any system we’ve designed.

For the above test, we used Deform/Colander and Pyramid.

While working on a project, I ran across a number of emails that didn’t properly validate using Colander. Digging into Colander’s code, the regexp used was rather basic. Chris McDonough confirmed this and said he would welcome a newer regexp.

However, FormEncode’s email validation also allows one to optionally check the DNS to see if there is a valid A or MX record – validation that might be handy. Additionally, we need to do credit card verification which is not currently included in Colander. The quick solution is to use FormEncode’s validation routines through Colander.

Our Form Schema:

def email_validate(address):
    try:
        valid = formencode.validators.Email().to_python(address)
        return True
    except Exception as message:
        return unicode(message)

class EmailTestSchema(colander.Schema):
    email = colander.SchemaNode(colander.String(),
        widget = deform.widget.TextInputWidget(size=60),
        validator = colander.Function(email_validate),
    )

Our view:

    @action(renderer='email_test.mako')
    def test(self):
        schema = EmailTestSchema()
        form = deform.Form(schema, buttons=('Add Email',))
        if self.request.POST:
            try:
                appstruct = form.validate(self.request.POST.items())
            except deform.ValidationFailure, e:
                return {'form':e.render()}

        return {'form':form.render()}

Now, our email validation uses FormEncode which contains some fairly detailed error messages which are passed through Colander.

After more coding, I’ll post another solution that might answer some other issues I ran into with email validation.

My first attempt at migrating a Pylons project to Pyramid was accomplished without too much difficulty. That project was relatively small, however, this project hasn’t been put into production.

A brief history of this application:

We have a legacy PHP application which consists of 78k lines of code. Originally we started rewriting the application in Turbogears 2.0 and later moved over to Pylons with ToscaWidgets. We used ToscaWidgets because all of our forms had been written for TG2. The application never made it into production, but, was substantial enough that we felt it would be a good test to convert it to Pyramid since we’re focusing our development efforts on Pyramid.

Initially I started to write a script to do the migration from Pylons to Pyramid and from ToscaWidgets to Deform but abandoned that. Changing the controllers from Pylons to Pyramid was fairly easy.

Most of the pylons imports were commented out and replaced with:

import deform
import colander

from pyramid.response import Response
from pyramid.view import action
from pyramid.security import authenticated_userid
from pyramid.threadlocal import get_current_request
import webhelpers.paginate as paginate

References to tmpl_context. were altered, our __before__ action for authkit was removed and replaced with an __init__ since we were using handlers. Routes were modified and given unique names, templates were modified to remove references to ${h. and ${tmpl_context., and we began the process of stepping through the application. Initially I had written a script to convert the ToscaWidgets form models over to Deform schemas, but, after two hours, it became obvious it would take more time to do that than to manually recreate the forms. After working through much of the process, I’m debating whether this project should have been migrated over to FormAlchemy as almost every form is a duplicate of the SQL schema. Since it isn’t in production, we still have some time to make that decision.

What follows is a summary of the errors received and what caused the errors.

TypeError: ‘NoneType’ object is not iterable

I ran into this while importing some routes from an existing project. Since routes must be uniquely named in Pyramid, a route with the same name will replace a prior route. Some of the route names have gotten quite unwieldy.

TypeError: object.__new__() takes no parameters

Solution, add the __init__ block to your class in your handler.

class YourClass(object):
    def __init__(self, request):
        self.request = request

ValueError: Non-response object returned from view named (and no renderer): {‘template’: ‘billing_index’}

Turbogears 2.0 code:

    def index(self, **kw):
        return dict(template='billing_index')

modify to:

    @action(renderer='billing_index.mako', permission='client')
    def index(self, **kw):
        return {}

AttributeError: ‘Undefined’ object has no attribute ‘form’

Pyramid doesn’t pass the tmpl_context., c. or h. through to the template, so, the return values of each of the actions needs to pass the fields required. As a result, any template code that replies on these globals needs to be modified, and the corresponding action needs to return the values in the dictionary. You can use one of the pylons templates which will instantiate a subscriber method to replicate the Pylons globals if you want.

AttributeError: ‘Undefined’ object has no attribute ‘literal’

Existing forms are relying on the pylons h. global.

${h.literal(form(value=value))}

UnboundLocalError: local variable ‘clients’ referenced before assignment

This occurs when you remove tmpl_context. and are left with a variable assigned that matches the class. For example:

tmpl_context.clients = meta.Session.query(clients).filter_by(client_id==1).all()

When tmpl_context. is removed, the class clients is cast incorrectly.

AttributeError: ‘Undefined’ object has no attribute ‘pager’

Paginate itemset isn’t being passed to the template.

TypeError: ‘NoneType’ object is not iterable

Paginate is getting the result set, rather than the paginate set passed in the return dictionary.

NameError: global name ‘request’ is not defined

request.matchdict/request.params -> self.request.matchdict/self.request.params

In a handler, request. is referred to as self.request. If you’ve converted things over to a handler rather than writing the individual routes for each action, you’ll need to preface any request. with self.

NotImplementedError: no URL generator available

Webhelpers pagination doesn’t know how to generate URLs in Pyramid, but, you can use a callable to generate the URLs. This requires access to pyramid.threadlocal which is generally not recommended, but, does allow you to use paginate until a pyramid compatible paginate is written.

The generator you need looks like this:

from webhelpers.util import update_params
from pyramid.threadlocal import get_current_request

def get_page_url(**kw):
    return update_params(get_current_request().path_info, **kw)

In your paginate block, you need to add the following:

url=get_page_url,

        paginator = paginate.Page(
            features,
            page=int(self.request.params.get('page', 1)),
            items_per_page = 40,
            url=get_page_url,
        )

RuntimeError: Caught exception rendering template. TypeError: ‘int’ object is not iterable

deform select widget requires tuple for the dropdown creation. Toscawidgets would build the right hand side if it was passed a list of IDs.

TypeError: ‘Undefined’ object is unsubscriptable

Returning a dict, removing tmpl_context., a ${value['asdf']} is missing ‘value’:value being passed in the return dict.

TypeError: sequence item 10: expected string or Unicode, Undefined found

When using deform, return {‘form’:form.render()} rather than return {‘form’:form}.

Summary

This is the second application I’ve converted from Pylons 1.0 to Pyramid and most of the issues have been syntax issues. Hopefully the error summary above will save someone some time.