7. Functions

Writing functions using def is a cornerstone of all programs. In this tutorial we will look into some of the advance usage of functions viz.

1. closures
2. callback-functions
3. control-flow
4. keyword-only arguments
5. default-arguments
6. annotations
7. any-number-of-arguments (*args & **kwargs) etc.

7.1: Function with any number of arguments (*args, **kwargs)

Problem

You want to write a function that accepts any number of arguments.

Solution

For a function that accepts any number of positional arguments use a * argument.

For a function that accepts any number of keyword arguments use a ** argument.

To write any function that accepts any numbe rof positional arguments, use a * argument as shown below. In this example, rest is a tuple of all extra positional arguments passed. In above code, it is treated as a sequence in further calculations inside the function.

def avg(first, *rest):
    return (first + sum(rest)) / (1 + len(rest))

## ------------------------------------------------- ##

avg(1, 2, 3)        ## return=2.0; first=1, rest=(2, 3)
avg(1, 2, 3, 4)     ## return=2.5; first=1, rest=(2, 3, 4)

To accept any number of keyword arguments use a ** argument as shown below. In the below code, attrs is a dictionary that holds the passed keyword arguments (if any)

import html

def make_element(name, value, **attrs):
    keyvals = [f"{key}='{val}'" for key, val in atts.items]
    attr_str = "".join(keyvals)
    element = f"<{name}{attr_str}>{html.escape(value)}</{value}>"
    return element

## Creates: "<item size='large' quantity=6>Albatross</item>"
make_element("item", "Albatross", size="large", quantity=6)

## Creates: "<p>&lt;spam&gt;</p>"
make_element(p, "<spam>")

If you want a function that accepts both any-number-of-positional-arguments aswell as any-number-of-keyword-arguments use both * & ** arguments as shown below

def anyargs(*args, **kwargs):
    print(args)     ## a tuple
    print(kwargs)   ## a dictionary

Discussion

A * argument can only appear as the last positional argument in a function and a ** argumnet can only appear as the last argumnet in a function.

A subtle aspect of function definition is that arguments can still appear after a * argument by they are treated as keyword argument only (as discussed in further lessons).

## valid function definitions
def a(x, *args, y): pass
def b(x, *args, y, **kwargs): pass


## WRONG function defintions
## here argument "z" appears after `**kwargs`; hence its a WRONG definition

def c(x, *args, **kwargs, z): pass

7.2: Keyword-only arguments

Problem

You want to write a function that accepts only keyword arguments.

Solution

This feature is easy to implement if you place the keyword argument after the * argument or a single unnamed * as shown below.

def recv(maxsize, *, block):
    """Receives a message"""
    pass

##-------------------------------------##

recv(1024, True)            ## Type Error
recv(2024, block=True)      ## OK

This technique can also be used to specify keyword arguments for functions that accept varying numbe rof positional arguments as shown below.

def minimum(*values, clip=None):
    m = min(values)
    if clip is not None:
        m = clip if clip > m else m
    return m

## ----------------------------------------- ##

minimum(1, 2, 5, -5, 10)            ## -5
minimum(1, 2, 5, -5, 10, clip=0)    ## 0

Discussion

Keyword-only arguments are a good way to enforce greater code-clarity when specifying optional function arguments. For example consider the following code:

msg = recv(1024, False)

If someone is not familiar with recv (written in above section), they might not know what does False mean in this scenario. On the other hand, its much clearer to the user if someone writes the call as below:

msg = recv(1024, block=False)

The use of keyword-only arguments is often very useful for tricks involving **kwargs, since they show up nicely when the user asks for help()

>>> help(recv)
Help on function recv in module __main__:
recv(maxsize, *, block):
    """Receives a message"""

Keyword-only arguments also have utility in advance usage like argument injection using *args & **kwargs.

7.3: Function argument annotations

Problem

You've written a function but you want add some additional information about the arguments so that users can know more easily how a function is supposed to be used.

Solution

Function argument annotations can be very useful in givin gprogrammers hint about how a function is supposed to be used. Cosider the following annotated function.

## normal function definition
def add(x, y):
    return x + y

## Annotated function definition -- MUCH BETTER for users
def add(x:int, y:int) -> int:
    return x + y

The Python Interpreter does not add any semantic meaning to the attached annotations. Neither are they type-cheked nor does Puython behave any differently than before. However, they might give useful hunts about types that would be useful for users or third-party libraries.

Any type of object can be uased as an annotation (eg. numbers, string, instances, types, etc.).

Discussion

Function annotations are merely stored in function's __annotations__ attribute.

>>> add.__annotations__
{"y": <class "int">,
 "return": <class "int">,
 "x": <class "int">,
}

Although, annotations have many potential use-cases, their primary utility is probably just documentation. Because Python does not have type decalrations, it can be difficult to read a source code without much documentation and hind about the types of objects; function annotations are one way to resolve this problem.

More advance usage of function annotations is to implement multiple dispatch (i.e. overloaded functions)

7.7: Capturing variables in Anonymous functions

Problem

You have defined an anonymous function using lambda and but you also need to capture the value of certain variable at the tim eof definition.

Solution

Consider the behaviour of the following code a python console:

>>> x = 10
>>> a = lambda y: x + y
>>>
>>> x = 20
>>> b = lambda y: x + y

Now ask yourself a question. What is the value of a(10) and b(10) now? If you think that the values will be 20 and 30; you would be wrong. The python console gives the follwoing results.

>>> a(10)
30
>>>
>>> b(10)
30

The problem here is that x used here (in the lambda expression) is a free variable; that gets bound at runtime, not at definition time. The value x in the lambda expression is whatever the value of x variable happens to be at the time of execution of the statement a(10), b(10). For example see below code:

>>> x=15
>>> a(10)
25
>>>
>>> x=3
>>> a(10)
13

If you want an anonymous lambda function to capture a value at the point of definition include that value as a default value in the lambda expression definition as follows.

>>> x = 10
>>> a = lambda y, x=x: x + y
>>>
>>> x = 20
>>> b = lambda y, x=x: x + y
>>>
>>> a(10)
20                          ## Note the difference here from above code (earlier this was 30)
>>>
>>> b(10)
30                          ## Note that "b" captured the correct value of "x"

Discussion

The problem addressed here comes up very often in code that tries to be a just a bit more clever. For example, creating a list of lambda expressions using a list comprehension or a loop of some kind and expecting the lambda expressions to remember the iteration variable at the time of defintion. For example:

>>> funcs = [lambda x: x+n for n in range(5)]
>>> for f in funcs:
...     print(f(0))
...
4
4
4
4
4
>>>

Notice, how all lambda expressions take the same final value of n from the range(5) expression.

But, if we specifically captuer the value of n using default arguments, then we get the behaviour we want

>>> funcs = [lambda x, n=n: x + n for n in range(5)]
>>> for f in funcs:
...     print(f(0))
...
0
1
2
3
4
>>>

7.8: functools.partial()

Making an N-argumnet callable work asa callable with fewwer argumnets

You have a callable that you want to make it work with other Python code. possbly as a callabel or handler, but it takes too many argumnets and causes an exception when called.

Solution

If you need to reduce the number of arguments to a function, you should use functools.partial() The partial() function allows you to assign fixed values to one or more of the arguments thus reduce the number of required argumnets in subsequent calls of the reduced function. as shown below.

def spam(a, b, c, d):
    print(a, b, c, d)

Now consider functools.partial() to fix certain arguments.

>>> from functools import partial
>>> 
>>> s1 = partial(spam, 1)               ## a=1
>>> s1(2,3,4)                           ## note that it takes only 3 arguments
1 2 3 4

>>> s2 = partial(spam, d=42)            ## d=42
>>> s2(1,2,3)
1 2 3 42
>>> s2(5, 99, 32)
5 99 32 42

>>> s3 = partial(spam, 1, 2, d=42)      ## a=1, b=2, d=42
>>> s3(99)
1 2 99 42
>>>

Discussion

...

7.9: Replacing single method classes with functions

Problem

You have a class that only defines a single method except __init__() method. However, to simplify your code, you would much rather have a simple function.

Solution

In many cases, single method classes can be turned into functions using closures.

Consider the following example, which allws users to fetch URLs using a kind of template-scheme.

from urllib.request import urlopen

class UrlTemplate:
    def __init__(self, template):
        self.template = template
    def open(self, **kwargs):
        return urlopen(self.template.format_map(kwargs))

## -------------------------------------------------------------------------------- ##

## Usage of above class
## Example Use: Download stock data from yahoo

yahoo = UrlTemplate("http://finance.yahoo.com/d/quotes.csv?s={names}&f={fields}")
for line in yahoo.open(names="APPL, FB, TSLA", fields="sl1c1v"):
    print(line.decode("utf-8"))

This URLTemplate class be very easily replaced with a much simpler function using closures. as shown below.

## Using closures
from urllib.request import urlopen

def url_template(template):
    def opener(**kwargs):
        return urlopen(template.format_map(kwargs))
    return opener

## ------------------------------------------------------------------------------- ##

## Example Usage of the above CLOSURE "url_template"

yahoo = url_template("http://finance.yahoo.com/d/quotes.csv?s={names}&f={fields}")
for line in yahoo(names="APPL, FB, TSLA", fileds="sl1c1v"):
    print(line.decode("utf-8"))

7.10: Carrying extra state with callback functions

Problem

You are writing code that relies on the use of callback functions (e.g. event handlers, completion callback, etc.); but you want to have the callback function carry extra state to be used inside the callback function itself.

Solution

Let's first talk about callback functions which are used in asynchronous processing via the following example:

def apply_async(func, args, *, callback):
    ## compute the result
    result = func(*args)

    ## invoke the callback
    callback(result)

In reality, such code might do all sorts of advanced processing involving threads, processes, & timers. But here we are just focusing on the invocation of the callback. Below, we show how to use the above code:

def print_result(result):
    print(f"Got: {result}")

def add(x, y):
    return x + y

>>> apply_async(add, (2, 3), callback=print_result)
Got: 5
>>> apply_result(add, ('Hello", "World'), callback=print_result)
Got: "HelloWorld"

As you can notice above, the callbakc function print_result() only accepts a single argument i.e result. No other information is provided. This lack of information can be an issue when we want our callback function to interact with other contextual variables or other parts of the code. Below we provide some probable solutions to handle this issue:

Some ways to carry extra information with callbacks:

1. Use bound methods instead of regular sinple functions.
2. Use closures
3. Use coroutines.
4. Using functools.partial()

Using BOUND METHODS: One way to carry extra information in a callback is to use bound methods instead of simple function. For example, this below class keeps an internal sequence number that is incremented everytime a result is received.

class ResultHandler:
    def __init__(self):
        self.sequence = 0
    def handler(self, result):
        self.sequence += 1
        print(f"[{self.sequence}] Got: {result}")

To use this class ResultHandler, we need to create its instance and use its handler() method as a callback.

>>> rh = ResultHandler()                 ## creating an instance of the class
>>> apply_async(add, (2, 3), callback=rh.handler)
[1] Got: 5
>>> apply_async(add, ("Hello", "World"), callback=rh.handler)
[2] Got: "HelloWorld"

Notice in the above output, how the instance of ResultHandler class rh uses its internal state variable self.sequence across two different calls of the callback function rh.handler.

---

Using CLOSURES: As an alternative we can also use closures to embed additional information in the callback function.

def make_handler():
    sequence = 0
    def handler(result):
        nonlocal sequence
        sequence += 1
        print(f"[{sequence}] Got: {result}")
    return handler

>>> handler = make_handler()                    ## our callback function to be used
>>> apply_async(add, (2,3), callback=handler)
[1] Got: 5
>>> apply_async(add, ("Hello", "World"), callback=handler)
[2] Got: "HelloWorld"                                       ## the callback remembers the count sequence

---

Using COROUTINES: Sometimes we can also use a coroutine to accomplish what we achieved using a bound method or a closure.

## Using COROUTINES to add context to a callback function

def make_handler():
    sequence = 0
    while True:
        result = yield
        sequence += 1
        print(f"[{sequence}] Got: {result}")

We can use it as usual as before.

>>> handler = make_handler()                            ## our callback function
>>> apply_async(add, (2, 3), callback=handler.send)     
[1] Got: 5
>>> apply_async(add, ("Hello", "WORLD!!"), callback=handler.send)
[2] Got: "HelloWORLD!!"

NOTICE the use of send to push value to the yield in the callback coroutine

---

Using functools.partial(): As an alternative, we can also carry state into a callback using an extra argument and a partial function application.

class SeqeunceNo:
    def __init__(self):
        self.sequence = 0

def handler(result, seq):
    seq.sequence += 1
    print(f"[{seq.sequence}] Got: {result}")

>>> from functools import partial
>>> seq = SequenceNo()
>>> apply_async(add, (2,3), callback=partial(handler, seq=seq))
[1] Got: 5
>>> apply_async(add, ("HELLO_", "WORLD!!!"), callback=partial(handler, seq=seq))
[2] Got: "HELLO_WORLD!!!"

---

Because, lambda expressions can be very useful in creating partial functions. We can use lambda expressions (with added contextual argumnets) as callbacks as shown below.

>>> seq = SequenceNo()
>>> apply_async(add, (2,3), callback=lambda r: handler(r, seq))
[1] Got: 5

Discussion

Software based on callback functions often runs the risk of turning into a huge tangled mess.

Part of the reason is that callback function is often disconnected from the code that made the initial request leading up the callback execution". Thus the execution environment between making the request and handling the result is effectively lost.

If you want the callback function to continue with a procedure involving multiple steps, you have to figure out how to save and restore the associated states.

There are really two main promising ways to capture and restore context states:

1. You can carry it around in a class instance. Attach it to bound method perhaps.
2. Or you can carry it around in a closure (an inner function).

Of the above two techniques, probably the closures approach is more natural and lightweight in that they are built from functions. They also automatically captures all the variables used.

But, if using closures, we need to pay careful attention to mutable variables. In the above solution we use nonlocal declaration for variable sequence to specify that we were using the variable from withing the closure; otherwise we will get an ERROR.

---

The use of coroutines is interesting and very closely resembles the usage of closures.

In some sense, it is even cleaner as we don't have to worry about the nonlocal declarations.

Only potential downsides of using coroutines is that its somewhat difficult to understand conceptually and we have to remembr some nitty-gritty details of its usage (for example, making sure to call next() before using the couroutine as a callback etc...)

Nevertheless, coroutines are very useful in defining inlined callbacks.