Python Course
Session 7 – Decorators
Python has an advanced language feature called decorator which is a little like C macros but much more like Lisp macros. A decorator is easy to use but tricky to construct. The usage side is sufficiently simple that we can cover this in the introduction.
There are function decorators, and class decorators. A decorator can only be applied to a callable, but note that a class with an __call__ method is in fact callable.
Here is a decorator timeit applied to a function:
@timeit
def fn():
...
The timeit decorator wraps the function call with before and after code which will then be executed in addition to the original function's code, whenever the function is called. And timeit is written as a function, which returns a function definition:
def timeit(function_to_decorate):
def some_function():
...
return some_function
Python effectively transforms:
@timeit
def fn():
...
to:
def fn():
...
fn = timeit(fn)
The @timeit is therefore just a little “syntactic sugar”.
The overall effect of fn = timeit(fn) is that the original fn function is replaced with a different function constructed by the timeit function.
The remainder of this session of the course is on how to write a function such as timeit!
BTW you can stack decorators (invoked sequentially bottom up) and they can take arguments, e.g.:
@timeit
@memoize(limit=10)
def fn():
...
You need a good understanding of variable scope, first class functions and closures.
In Python, a variable is a name for an object. The object exists independently from the name, and in fact may have more than one name. If all names for an object are deleted or go out of scope, rendering the object unreferenceable, then the object will be considered for garbage collection, i.e. deletion.
A global variable is one which is defined at the top level of a module. A local variable is one defined within the current (non-top-level) scope.
You start a new scope when you go into a function or class. Nested function definitions each start a new scope.
If you define a variable in a new scope, that variable is separate from any variable with the same name in an outer scope:
x = 1
def fn():
x = 2
# Here, x is 2 and this x is a different x from the outer one.
fn()
# Here, x is still 1
Provided you don't define a separate variable, Python allows access to an outer variable:
x = 1
def fn():
z = x
# Here, z is 1
# But, it would be an error to now try to define an x here.
fn()
# Here, z in inaccessible
You can “adjust” the scope rules through use of the global statement:
x = 1
def fn():
global x
x = 2
# Here, x is 2 and this x is the outer x.
fn()
# Here, x is 2.
And through use of the nonlocal statement:
x = 1
def fn():
x = 2
def fn2():
nonlocal x
x = x + 1
# Here, x is 3 and this x is the “closest” outer x.
fn2()
# Here, x is 3.
fn()
# Here, this x is 1
In Python, a function is a first class object. What we're talking about here is the uncalled object. It can be treated like any variable. In particular, it can be: stored; passed as a parameter to a function; returned from a function.
This is important to the construction of a decorator, because it can need all of those things.
The construction of a decorator generally involves defining functions within functions and returning such definitions. Here's an example of a function defining and returning a function:
def fn():
def fn2():
return 1
return fn2
fn2 = fn()
x = fn2()
# Here, x is 1
Let's add a variable:
def fn():
x = 1
def fn2():
return x
return fn2
fn2 = fn()
x = fn2()
# Here, x is 1
This might not seem like anything special, but Python has actually worked hard to achieve this: the value of the variable x has been specially remembered for the returned fn2, so that when fn2 is ultimately called, it can execute the “return x” and substitute in the remembered value of x.
This is called a closure, where values of variables necessary for the ultimate execution of a function object are preserved as part of that function object, even though those variables are no longer in scope.
As we saw in the Introduction, a decorator is a function that takes one function and returns another function. It's sensible to derive the returned function from the one that is passed in. The decorator doesn't modify that passed in function per se, but it does wrap it with before and after code.
So, let's build a timeit decorator. As we saw:
@timeit
def fn():
...
effectively means:
def fn():
...
fn = timeit(fn)
So the timeit function needs to look like:
def timeit(function_to_decorate):
def wrapper():
somecode
return wrapper
I.e. we define a function inside timeit, and return that as a replacement for function_to_decorate.
The somecode needs to do the timing, as well as call function_to_decorate and return its returned value:
def timeit(function_to_decorate):
def wrapper():
start_time = time.clock()
retval = function_to_decorate()
duration = time.clock() - start_time
print('Time taken:', "{:.9f}".format(duration), 'seconds')
return retval
return wrapper
That's our basic timeit decorator. It should really be generalised so it can be applied to a function that has arguments, thus:
def timeit(function_to_decorate):
def wrapper(*args, **kwargs):
start_time = time.clock()
retval = function_to_decorate(*args, **kwargs)
duration = time.clock() - start_time
print('Time taken:', "{:.9f}".format(duration), 'seconds')
return retval
return wrapper
And, what if we wanted to apply it to a recursive function? We don't really want to see time-taken reports for each call. Rather, we just want to time the top-level call. It's possible:
def timeit(function_to_decorate):
started_timing = False
def wrapper(*args, **kwargs):
nonlocal started_timing
if started_timing:
return function_to_decorate(*args, **kwargs)
started_timing = True
start_time = time.clock()
retval = function_to_decorate(*args, **kwargs)
duration = time.clock() - start_time
print('Time taken:', "{:.9f}".format(duration), 'seconds')
started_timing = False
return retval
return wrapper
and that works well for e.g.:
@timeit
def f(n):
if n >= 0: f(n-1)
f(100)
Note: This still isn't “perfect” and would benefit from some try-finally code to wind up timing in the event of an exception being propagated up from the call to function_to_decorate.
Here's the same timeit, but implemented as a class. Implementing as a class neatly avoids having to define a wrapper function, and there is no change to the @timeit usage.
import time
class timeit():
def __init__(self, function_to_decorate):
self.function_to_decorate = function_to_decorate
self.started_timing = False
def __call__(self, *args, **kwargs):
if self.started_timing:
return self.function_to_decorate(*args, **kwargs)
self.started_timing = True
start_time = time.clock()
retval = self.function_to_decorate(*args, **kwargs)
duration = time.clock() - start_time
print('Time taken:', "{:.9f}".format(duration), 'seconds')
self.started_timing = False
return retval
Decorators themselves can have arguments, e.g.: suppose we only want to report on a function call that takes greater than a certain amount of time:
@timeit(mintime)
The way the class is invoked is very different from the no-argument case, because Python first invokes the timeit(mintime) call, then decorates the result. We need a use a “wrapper” function:
import time
class timeit():
def __init__(self, mintime):
self.mintime = mintime
self.started_timing = False
def __call__(self, function_to_decorate):
def wrapper(*args, **kwargs):
if self.started_timing:
return function_to_decorate(*args, **kwargs)
self.started_timing = True
start_time = time.clock()
retval = function_to_decorate(*args, **kwargs)
duration = time.clock() - start_time
if duration > self.mintime:
print('Time taken:', "{:.9f}".format(duration), 'seconds')
self.started_timing = False
return retval
return wrapper
@timeit(0.000025)
def f(n):
if n >= 0: f(n-1)
If you implement a decorator-with-arguments as a function, you will need an extra wrapping layer for that too!
When you decorate a function, the original function is replaced by a wrapper function. These two are different functions, with different attributes such as the doc string. In some cases this may not be important but it can be fixed. For decorators implemented as functions, decorate the wrapper function with the wraps decorator from the functools module:
@functools.wraps(function_to_decorate)
def wrapper(*args, **kwargs):
...
For no-argument decorators as classes, a different “fix” is needed by using update_wrapper from functools:
def __init__(self, function_to_decorate):
functools.update_wrapper(self, function_to_decorate)
...
Decorators are appealing because they make it easy to apply certain pre-canned functionality to existing functions, without editing the function code. The Python Decorator Library provides many decorators.
Some general uses for decorators are:
•Timing a function
•Counting function calls
•Logging function calls
•Tracing specific functions
•Memoising functions
•Timing out a function call
Python 3 Patterns, Recipes and Idioms - Decorators
Mark Lutz's book: Learning Python. Very comprehensive.