Python Metaclasses (Metaproramming)

Introduction

Metaprogramming involves programs treating themselves as data. This implies they can introspect/generate/modify themselves. There are two broad classes of metaprogramming. One is introspection-oriented, where the focus is on inspecting or modifying programming language features. The other is code-oriented, where code is treated as data that can be dynamically generated or modified. The aspect of metaprogramming discussed here is introspection-oriented, and is the idea of a metaclass in Python.

Classes in Python are instances of the class type, which defines how all ordinary classes are created. All Python classes inherit from the class object. type defines how classes are created, and thus defines how the class object is created. But since type is also a class, it inherits from object, implying isinstance(object, type) = isinstance(type, object) = isinstance(type, type) = instance(object, object) = True. How marvelous.

The familiar class definition structure

class Foo:
    pass

is equivalent to

Foo = type("Foo", (), {}) # I.e. type(name, bases, namespace).

where name is the __name__ attribute of the defined class; bases is a tuple of base classes used to define the class's method resolution order, returned by its __mro__ attribute; and namespace is where all class attributes and methods are stored, and is required to be a subclass of dict. When the class is finally created, namespace is copied and can no longer be modified directly. The __dict__ attribute of a class returns a mappingproxy instance, which is a read-only view of the copied namespace. (Attributes and methods stored in the namespace can still be modified, just not by a direct manipulation of the underlying copied namespace. That's why class attributes and methods can be set/modified.)

A metaclass is simply a class used to define how classes are created. Hence, type is a metaclass, and infact is the base metaclass. Python allows the creation of custom metaclasses by subclassing type. This approach can be used to enhance classes beyond what class decorators can do, and it has the added benefit of being inheritable (unlike class decorators).

Metaclass structure

The basic structure of a metaclass is:

class Metaclass(type):
    def __new__(mcs, name, bases, namespace, **kwargs):
        return super().__new__(mcs, name, bases, namespace)

    @classmethod
    def __prepare__(mcs, name, bases, **kwargs):
        return super().__prepare__(name, bases, **kwargs)

    def __init__(cls, name, bases, namespace, **kwargs):
        super().__init__(name, bases, namespace, **kwargs)

    def __call__(cls, *args, **kwargs):
        return super().__call__(*args, **kwargs)

A class could then use this metaclass as follows:

class User(metaclass=Metaclass [, kwarg=val, ...]):
    ...

• __prepare__ returns the namespace for the new class.

• __new__ creates and returns the new class, using the namespace returned by __prepare__. It is a static method, but can be used without the @staticmethod decorator.

• __init__ does some initializaion on the class returned by __new__.

• __call__ is executed when the class is called to instantiate an instance. In this case, user = User(<init args>) -> the metaclass __call__ is executed before User.__init__ is called.

Something worth mentioning is that if implementation-specific methods are added to a metaclass, they are to have the form:

class Metaclass(type):
    ...  # Same as before.

    def custom_method(mcs, *args, **kwargs):
        ...

    def other_custom_method(mcs, *args, **kwargs):
        # How to use `custom_method`:
        other_args = (arg1, ...)
        other_kwargs = {key1: val1, ...}
        result = mcs.custom_method(mcs, *other_args, **other_kwargs)

Keyword arguments after metaclass=Metaclass in the User class definition are stored in the kwargs dict for __prepare__, __new__, and __init__.

Thus, running the code file with the above structure defined in it leads to the execution of methods in the order Metaclass.__prepare__ -> Metaclass.__new__ -> Metaclass.__init__. user = User(<init args>) then calls in order Metaclass.__call__ -> User.__new__ -> User.__init__.

NOTE: Since Python doesn't enforce return types for the above methods, care should be taken to avoid hard to debug errors. For example, returning a different class in Metaclass.__new__ could lead to painful to track bugs.

The power and dangers of using metaclasses can be illustrated using the following example. (NOTE that this is just an illustration of the 'magic' allowed by metaclasses, not an attempt to build something exactly useful.)

Metaclass example

To gain the most value from this section, I suggest that you open your favorite code editor and code along, testing and playing with the code to fully understand how things work.

The first idea is to enforce the parameter annotations (if any) of the __init__ method, and also to ensure that any attribute set on an instance of a class matches the class annotation for that attribute (if any). The first code section is the following:

from inspect import signature


class _Untyped: ...


def _get_cls_attr_annotation(cls, attr):
    for scls in cls.__mro__:
        if attr in getattr(scls, "__annotations__", ()):
            return scls.__annotations__[attr]
    return _Untyped


def _check_attr_matches_cls_annotation(cls, attr, val):
    expected_type = _get_cls_attr_annotation(cls, attr)
    if not (expected_type is _Untyped or isinstance(val, expected_type)):
        raise TypeError(f"Expected type {expected_type} for attribute {attr}")


def get_func_annotations(signature):
    return {
        arg: param.annotation
        for arg, param in signature.parameters.items()
        if not param.annotation is signature.empty
    }

Key elements are:

• A 'marker' class _Untyped is used to represent the type of a parameter/attribute with no annotation.

• _get_cls_attr_annotation goes through the method resolution order of a class to find the first annotation for an attribute, if it exists. If it doesn't, it returns _Untyped. Stopping at the first match on the way up the method resolution order ensures that a subclass annotation takes precedence over that of its superclass.

• _check_attr_matches_cls_annotation checks if an attribute's value matches that of the class annotation returned by _get_cls_attr_annotation. TypeError is raised if this is not the case.

• get_func_annotations returns a dictionary of parameter annotations for a function, obtained through the Signature object of the function, which is returned by calling signature from the inspect module, with the function as an argument.

Next, we create a metaclass _TypedMeta and a base class Typed just to make using the metaclass a bit cleaner. (Classes using the metaclass only have to subclass Typed). The metaclass has the following key elements:

• The __new__ method creates the class and attaches a custom __setattr__ which is called when an attribute is set on its instance. It checks if the value provided for the attribute matches its class annotation, stopping at the first annotation match for a class in the class's method resolution order. This implies that for the same attribute, a subclass annotation takes precedence over that of its superclass.

• The __call__ method binds arguments provided to the class's __init__ method for object initializaion (excluding self since the instance of the class hasn't been created at this point). All values not matching the annotated expected types value are collected into a dictionary, which is used to format a raised TypeError exception.

class _TypedMeta(type):
    def __new__(mcs, name, bases, namespace):
        cls = super().__new__(mcs, name, bases, namespace)

        def set_attr(self, attr, val):
            def set_attr(self, attr, val):
            _check_attr_matches_cls_annotation(type(self), attr)
            object.__setattr__(self, attr, val)

        cls.__setattr__ = set_attr
        return cls

    def __call__(cls, *args, **kwargs):
        init_signature = signature(cls.__init__)
        annotations = get_func_annotations(init_signature)
        bound_args = init_signature.bind(
            # The class's `__new__` and `__init__` methods haven't been called.
            # There's no `self` yet, so we bind None to __init__'s `self` arg.
            None,
            *args,
            **kwargs,
        )
        mismatched_types = {}
        for arg, expected_type in annotations.items():
            if arg in annotations and not isinstance(
                bound_args.arguments[arg], expected_type
            ):
                mismatched_types[arg] = expected_type
        if mismatched_types:
            raise TypeError(
                f"Please provide expected type(s) for:\n\t{mismatched_types}."
                "\n({arg: <class 'ExpectedType'>, ...})"
            )
        return super().__call__(*args, **kwargs)



class Typed(metaclass=_TypedMeta): ...

There's no restriction on subclasses of Typed overriding the __setattr__ method provided by the metaclass's __new__ method, which would lead to instance attributes being allowed to be set to values not matching expected types. This should not be allowed to enforce consistent behaviour across subclasses. It would also be a good idea to extend method argument type checking to all methods in the class and also to enforce expected return types if the annotation exists, even for those added dynamically during runtime. To accomplish both, we add the following.

class OverrideDisallowed(Exception): ...


def _get_cls_attr_annotation(cls, attr):
    for scls in cls.__mro__:
        if attr in getattr(scls, "__annotations__", ()):
            return scls.__annotations__[attr]
    return _Untyped


def enforce_method_types(method):
    @wraps(method)
    def wrapper(*args, **kwargs):
        sig = signature(method)
        annotations = get_func_annotations(sig)
        bound_args = sig.bind(*args, **kwargs)
        mismatched_types = {}
        for arg, expected_type in annotations.items():
            if arg in annotations and not isinstance(
                bound_args.arguments[arg], expected_type
            ):
                mismatched_types[arg] = expected_type
        if mismatched_types:
            raise TypeError(
                f"Please provide expected type(s) for:\n\t{mismatched_types}."
                "\n({arg: <class 'ExpectedType'>, ...})"
            )
        # SILLY since we can only see the return type after running the method!
        return_val = method(*args, **kwargs)
        if (sig.return_annotation is not sig.empty) and not isinstance(
            return_val, sig.return_annotation
        ):
            raise TypeError(f"Expected return type {sig.return_annotation}.")
        return return_val

    return wrapper


class _RestrictedSetattrDict(dict):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._setattr_assigned = False

    def __setitem__(self, key, val):
        # Prevent overrides of '__setattr__' added by the metaclass.
        if key == "__setattr__":
            if not self._setattr_assigned:
                self._setattr_assigned = True
            else:
                raise OverrideDisallowed("Cannot override __setattr__")
        if callable(val):
            val = enforce_method_types(val)
        super().__setitem__(key, val)

Key elements are:

• A custom OverrideDisallowed exception.

• enforces_method_types is a method decorator which wraps a method and ensures that its annotated parameters and return value match expected types, or raises a TypeError if any of them don't. It's inefficient to check the return type in the way above since the method has to be run before the return value can be introspected.

• _RestrictedSetattrDict, a subclass of dict, raises an OverrideDisallowed exception if __setattr__ is a key in it there’s an attempt to update its value. It also checks if the value to be added for a key is a callable and decorates it with enforce_method_types before adding, if so.

Next we add a new function and update the metaclass:

class _TypedMeta(type):
    @classmethod
    def __prepare__(mcs, name, bases, **kwargs):
        return _RestrictedSetattrDict()

    def __new__(mcs, name, bases, namespace, **kwargs):
        # cls instance attribute setting.
        def set_attr(self, attr, val):
            _check_attr_matches_cls_annotation(type(self), attr)
            object.__setattr__(self, attr, val)

        namespace.__setitem__("__setattr__", set_attr)
        cls = super().__new__(mcs, name, bases, namespace)
        # Check if class attributes match annotations, if set.
        # cls's annotations take precedence over subclasses, hence
        # 'reversed(cls.__mro__)'.
        mismatched_types = {}
        annotations = {}
        for scls in reversed(cls.__mro__):
            if hasattr(scls, "__annotations__"):
                annotations.update(scls.__annotations__)
        for attr, type_ in annotations.items():
            if hasattr(scls.__dict__, attr) and not isinstance(
                scls.__dict__[attr], type_
            ):
                mismatched_types[attr] = scls.__dict__[attr]
        if mismatched_types:
            raise TypeError(
                f"Please provide expected type(s) for attrs:\n\t{mismatched_types}."
                "\n({class_arg: <class 'ExpectedType'>, ...})"
            )
        return cls

    def __setattr__(cls, attr, val):
        if callable(val):
            val = enforce_method_types(val)
        else:
            _check_attr_matches_cls_annotation(cls, attr, val)
        super().__setattr__(attr, val)

• The __prepare__ method returns a dictionary a _RestrictedSetattrDict instance.

• The __new__ method adds the custom __setattr__ to the class namespace. It also checks statically defined class attributes against their annotations, if any. For the same attribute, the annotation of a subclass of Typed takes precedence over its superclass. A TypeError is raised if that's not the case.

• __setattr__ on the metaclass is what is called whenever a class attribute is added to the class after the class has been created by the metaclass (i.e. it can be triggered even in the __new__ method of the metaclass). In other words, it is called when cls.attribute = value is executed.

The full code is thus:

from inspect import signature
from functools import wraps


class _Untyped:
    pass


class OverrideDisallowed(Exception):
    pass


def _get_cls_attr_annotation(cls, attr):
    for scls in cls.__mro__:
        if attr in getattr(scls, "__annotations__", ()):
            return scls.__annotations__[attr]
    return _Untyped


def _check_attr_matches_cls_annotation(cls, attr, val):
    expected_type = _get_cls_attr_annotation(cls, attr)
    if not (expected_type is _Untyped or isinstance(val, expected_type)):
        raise TypeError(f"Expected type {expected_type} for attribute {attr}")


def get_func_annotations(signature):
    return {
        arg: param.annotation
        for arg, param in signature.parameters.items()
        if not param.annotation is signature.empty
    }


def enforce_method_types(method):
    @wraps(method)
    def wrapper(*args, **kwargs):
        sig = signature(method)
        annotations = get_func_annotations(sig)
        bound_args = sig.bind(*args, **kwargs)
        mismatched_types = {}
        for arg, expected_type in annotations.items():
            if arg in annotations and not isinstance(
                bound_args.arguments[arg], expected_type
            ):
                mismatched_types[arg] = expected_type
        if mismatched_types:
            raise TypeError(
                f"Please provide expected type(s) for:\n\t{mismatched_types}."
                "\n({arg: <class 'ExpectedType'>, ...})"
            )
        # SILLY since we can only see the return type after running the method!
        return_val = method(*args, **kwargs)
        if (sig.return_annotation is not sig.empty) and not isinstance(
            return_val, sig.return_annotation
        ):
            raise TypeError(f"Expected return type {sig.return_annotation}.")
        return return_val

    return wrapper


class _RestrictedSetattrDict(dict):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._setattr_assigned = False

    def __setitem__(self, key, val):
        # Prevent overrides of '__setattr__' added by the metaclass.
        if key == "__setattr__":
            if not self._setattr_assigned:
                self._setattr_assigned = True
            else:
                raise OverrideDisallowed("Cannot override __setattr__")
        if callable(val):
            val = enforce_method_types(val)
        super().__setitem__(key, val)


class _TypedMeta(type):
    @classmethod
    def __prepare__(mcs, name, bases, **kwargs):
        return _RestrictedSetattrDict()

    def __new__(mcs, name, bases, namespace, **kwargs):
        # cls instance attribute setting.
        def set_attr(self, attr, val):
            expected_type = _get_cls_attr_annotation(type(self), attr)
            if not (expected_type is _Untyped or isinstance(val, expected_type)):
                raise TypeError(f"Expected type {expected_type} for attribute {attr}")
            object.__setattr__(self, attr, val)

        namespace.__setitem__("__setattr__", set_attr)
        cls = super().__new__(mcs, name, bases, namespace)
        # Check if class attributes match annotations, if set.
        # cls's annotations take precedence over subclasses, hence
        # 'reversed(cls.__mro__)'.
        mismatched_types = {}
        annotations = {}
        for scls in reversed(cls.__mro__):
            if hasattr(scls, "__annotations__"):
                annotations.update(scls.__annotations__)
        for attr, type_ in annotations.items():
            if hasattr(scls.__dict__, attr) and not isinstance(
                scls.__dict__[attr], type_
            ):
                mismatched_types[attr] = scls.__dict__[attr]
        if mismatched_types:
            raise TypeError(
                f"Please provide expected type(s) for attrs:\n\t{mismatched_types}."
                "\n({class_arg: <class 'ExpectedType'>, ...})"
            )
        return cls

    def __setattr__(cls, attr, val):
        if callable(val):
            val = enforce_method_types(val)
        else:
            _check_attr_matches_cls_annotation(cls, attr, val)
        super().__setattr__(attr, val)


class Typed(metaclass=_TypedMeta): ...

To illustrate the usage of the metaclass, we have:

# Example 1.
class User(Typed):
    # Throws TypeError since d is set to an int.
    d: str = 1


# Example 2.
class User(Typed):
    d: str

    def __init__(self, a: int, b, c: str):
        self.a = a
        self.b = b
        self.c = c
        self.d = 1  # TypeError.


# Throws TypeError since `self.d` is set to an int.
user = User(1, 2, 3)


# Example 3.
class User(Typed):
    d: str

    def __init__(self, a: int, b, c: str):
        self.a = a
        self.b = b
        self.c = c


user = User(1, 2, "3")  # Works.
user.a = "2"  # TypeError.
user.b = "3"  # Works.
user.c = 1  # TypeError.


# Example 4.
# class User(Typed):
   d: str = "a"

    def foo(self, x: int): -> int
        return "1"

# Class attributes.
User.d = 2  # TypeError.
User.d = "1"  # Works.

# Instance attributes.
user = User()
user.foo(1)  # Return value TypeError.
user.foo("1")  # Argument TypeError.

Caveats and final words

The above implementation gives odd results for property getters and setters, methods decorated with @classmethod, or descriptors in general. Function annotations for these aren't enforced. Methods decorated with @staticmethod behave the quirkiest, as illustrated below. Now let’s not begin to imagine the horrors of wrongly using a Typed class with multiple inheritance.

class User(Typed):
    @staticmethod
    def foo(x: int) -> int:
        return "1"

User.foo("2")  # Argument TypeError.
User.foo(2)  # Return value TypeError.
user = User()
user.foo()  # TypeError.
user.foo(2)  # Too many arguments provided to `sig.bind` in `enforce_method_types`.

This kind of 'magic' without extensive documentation could lead to a debugging nightmare when working with classes subclassing Typed. Also, metaclasses might be an overkill for simpler situations. Class decorators can be used for simpler cases, but they have the drawback of being uninheritable. The __init_subclass__ special method can be used to create inheritable class decorators. Metaclass usage is quite common in framework development. For example, it enables the automatic addition of several utility methods for Django models.

To see an even more exploratory idea of creating a 'typed' python environment, check out David Beazley's Pycon talk on metaclasses: https://youtu.be/sPiWg5jSoZI?si=e_bZwnlqhYiMe6ba.