# Python Metaclasses (Metaproramming) ## Introduction Metaprogramming involves programs treating themselves as data. This implies they can introspect/generate/modify themselves. There are two broad classes of metaprogramming. One is introspection-oriented, where the focus is on inspecting or modifying programming language features. The other is code-oriented, where code is treated as data that can be dynamically generated or modified. The aspect of metaprogramming discussed here is introspection-oriented, and is the idea of a metaclass in Python. Classes in Python are instances of the class `type`, which defines how all ordinary classes are created. All Python classes inherit from the class `object`. `type` defines how classes are created, and thus defines how the class `object` is created. But since `type` is also a class, it inherits from `object`, implying `isinstance(object, type) = isinstance(type, object) = isinstance(type, type) = instance(object, object) = True`. How marvelous. The familiar class definition structure ```python class Foo: pass ``` is equivalent to ```python Foo = type("Foo", (), {}) # I.e. type(name, bases, namespace). ``` where `name` is the `__name__` attribute of the defined class; `bases` is a tuple of base classes used to define the class's method resolution order, returned by its `__mro__` attribute; and `namespace` is where all class attributes and methods are stored, and is required to be a subclass of `dict`. When the class is finally created, namespace is copied and can no longer be modified directly. The `__dict__` attribute of a class returns a `mappingproxy` instance, which is a read-only view of the copied namespace. (Attributes and methods stored in the namespace can still be modified, just not by a direct manipulation of the underlying copied namespace. That's why class attributes and methods can be set/modified.) A metaclass is simply a class used to define how classes are created. Hence, `type` is a metaclass, and infact is the base metaclass. Python allows the creation of custom metaclasses by subclassing `type`. This approach can be used to enhance classes beyond what class decorators can do, and it has the added benefit of being inheritable (unlike class decorators). ## Metaclass structure The basic structure of a metaclass is: ```python class Metaclass(type): def __new__(mcs, name, bases, namespace, **kwargs): return super().__new__(mcs, name, bases, namespace) @classmethod def __prepare__(mcs, name, bases, **kwargs): return super().__prepare__(name, bases, **kwargs) def __init__(cls, name, bases, namespace, **kwargs): super().__init__(name, bases, namespace, **kwargs) def __call__(cls, *args, **kwargs): return super().__call__(*args, **kwargs) ``` A class could then use this metaclass as follows: ```python class User(metaclass=Metaclass [, kwarg=val, ...]): ... ``` * `__prepare__` returns the namespace for the new class. * `__new__` creates and returns the new class, using the namespace returned by `__prepare__`. It is a static method, but can be used without the `@staticmethod` decorator. * `__init__` does some initializaion on the class returned by `__new__`. * `__call__` is executed when the class is called to instantiate an instance. In this case, `user = User()` -> the metaclass `__call__` is executed before `User.__init__` is called. Something worth mentioning is that if implementation-specific methods are added to a metaclass, they are to have the form: ```python class Metaclass(type): ... # Same as before. def custom_method(mcs, *args, **kwargs): ... def other_custom_method(mcs, *args, **kwargs): # How to use `custom_method`: other_args = (arg1, ...) other_kwargs = {key1: val1, ...} result = mcs.custom_method(mcs, *other_args, **other_kwargs) ``` Keyword arguments after `metaclass=Metaclass` in the `User` class definition are stored in the `kwargs` dict for `__prepare__`, `__new__`, and `__init__`. Thus, running the code file with the above structure defined in it leads to the execution of methods in the order `Metaclass.__prepare__` -> `Metaclass.__new__` -> `Metaclass.__init__`. `user = User()` then calls in order `Metaclass.__call__` -> `User.__new__` -> `User.__init__`. NOTE: Since Python doesn't enforce return types for the above methods, care should be taken to avoid hard to debug errors. For example, returning a different class in `Metaclass.__new__` could lead to painful to track bugs. The power and dangers of using metaclasses can be illustrated using the following example. (NOTE that this is just an illustration of the 'magic' allowed by metaclasses, not an attempt to build something exactly useful.) ## Metaclass example To gain the most value from this section, I suggest that you open your favorite code editor and code along, testing and playing with the code to fully understand how things work. The first idea is to enforce the parameter annotations (if any) of the `__init__` method, and also to ensure that any attribute set on an instance of a class matches the class annotation for that attribute (if any). The first code section is the following: ```python from inspect import signature class _Untyped: ... def _get_cls_attr_annotation(cls, attr): for scls in cls.__mro__: if attr in getattr(scls, "__annotations__", ()): return scls.__annotations__[attr] return _Untyped def _check_attr_matches_cls_annotation(cls, attr, val): expected_type = _get_cls_attr_annotation(cls, attr) if not (expected_type is _Untyped or isinstance(val, expected_type)): raise TypeError(f"Expected type {expected_type} for attribute {attr}") def get_func_annotations(signature): return { arg: param.annotation for arg, param in signature.parameters.items() if not param.annotation is signature.empty } ``` Key elements are: * A 'marker' class `_Untyped` is used to represent the type of a parameter/attribute with no annotation. * `_get_cls_attr_annotation` goes through the method resolution order of a class to find the first annotation for an attribute, if it exists. If it doesn't, it returns `_Untyped`. Stopping at the first match on the way up the method resolution order ensures that a subclass annotation takes precedence over that of its superclass. * `_check_attr_matches_cls_annotation` checks if an attribute's value matches that of the class annotation returned by `_get_cls_attr_annotation`. `TypeError` is raised if this is not the case. * `get_func_annotations` returns a dictionary of parameter annotations for a function, obtained through the `Signature` object of the function, which is returned by calling `signature` from the `inspect` module, with the function as an argument. Next, we create a metaclass `_TypedMeta` and a base class `Typed` just to make using the metaclass a bit cleaner. (Classes using the metaclass only have to subclass `Typed`). The metaclass has the following key elements: * The `__new__` method creates the class and attaches a custom `__setattr__` which is called when an attribute is set on its instance. It checks if the value provided for the attribute matches its class annotation, stopping at the first annotation match for a class in the class's method resolution order. This implies that for the same attribute, a subclass annotation takes precedence over that of its superclass. * The `__call__` method binds arguments provided to the class's `__init__` method for object initializaion (excluding `self` since the instance of the class hasn't been created at this point). All values not matching the annotated expected types value are collected into a dictionary, which is used to format a raised `TypeError` exception. ```python class _TypedMeta(type): def __new__(mcs, name, bases, namespace): cls = super().__new__(mcs, name, bases, namespace) def set_attr(self, attr, val): def set_attr(self, attr, val): _check_attr_matches_cls_annotation(type(self), attr) object.__setattr__(self, attr, val) cls.__setattr__ = set_attr return cls def __call__(cls, *args, **kwargs): init_signature = signature(cls.__init__) annotations = get_func_annotations(init_signature) bound_args = init_signature.bind( # The class's `__new__` and `__init__` methods haven't been called. # There's no `self` yet, so we bind None to __init__'s `self` arg. None, *args, **kwargs, ) mismatched_types = {} for arg, expected_type in annotations.items(): if arg in annotations and not isinstance( bound_args.arguments[arg], expected_type ): mismatched_types[arg] = expected_type if mismatched_types: raise TypeError( f"Please provide expected type(s) for:\n\t{mismatched_types}." "\n({arg: , ...})" ) return super().__call__(*args, **kwargs) class Typed(metaclass=_TypedMeta): ... ``` There's no restriction on subclasses of `Typed` overriding the `__setattr__` method provided by the metaclass's `__new__` method, which would lead to instance attributes being allowed to be set to values not matching expected types. This should not be allowed to enforce consistent behaviour across subclasses. It would also be a good idea to extend method argument type checking to all methods in the class and also to enforce expected return types if the annotation exists, even for those added dynamically during runtime. To accomplish both, we add the following. ```python class OverrideDisallowed(Exception): ... def _get_cls_attr_annotation(cls, attr): for scls in cls.__mro__: if attr in getattr(scls, "__annotations__", ()): return scls.__annotations__[attr] return _Untyped def enforce_method_types(method): @wraps(method) def wrapper(*args, **kwargs): sig = signature(method) annotations = get_func_annotations(sig) bound_args = sig.bind(*args, **kwargs) mismatched_types = {} for arg, expected_type in annotations.items(): if arg in annotations and not isinstance( bound_args.arguments[arg], expected_type ): mismatched_types[arg] = expected_type if mismatched_types: raise TypeError( f"Please provide expected type(s) for:\n\t{mismatched_types}." "\n({arg: , ...})" ) # SILLY since we can only see the return type after running the method! return_val = method(*args, **kwargs) if (sig.return_annotation is not sig.empty) and not isinstance( return_val, sig.return_annotation ): raise TypeError(f"Expected return type {sig.return_annotation}.") return return_val return wrapper class _RestrictedSetattrDict(dict): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._setattr_assigned = False def __setitem__(self, key, val): # Prevent overrides of '__setattr__' added by the metaclass. if key == "__setattr__": if not self._setattr_assigned: self._setattr_assigned = True else: raise OverrideDisallowed("Cannot override __setattr__") if callable(val): val = enforce_method_types(val) super().__setitem__(key, val) ``` Key elements are: * A custom `OverrideDisallowed` exception. * `enforces_method_types` is a method decorator which wraps a method and ensures that its annotated parameters and return value match expected types, or raises a `TypeError` if any of them don't. It's inefficient to check the return type in the way above since the method has to be run before the return value can be introspected. * `_RestrictedSetattrDict`, a subclass of `dict`, raises an `OverrideDisallowed` exception if `__setattr__` is a key in it there’s an attempt to update its value. It also checks if the value to be added for a key is a callable and decorates it with `enforce_method_types` before adding, if so. Next we add a new function and update the metaclass: ```python class _TypedMeta(type): @classmethod def __prepare__(mcs, name, bases, **kwargs): return _RestrictedSetattrDict() def __new__(mcs, name, bases, namespace, **kwargs): # cls instance attribute setting. def set_attr(self, attr, val): _check_attr_matches_cls_annotation(type(self), attr) object.__setattr__(self, attr, val) namespace.__setitem__("__setattr__", set_attr) cls = super().__new__(mcs, name, bases, namespace) # Check if class attributes match annotations, if set. # cls's annotations take precedence over subclasses, hence # 'reversed(cls.__mro__)'. mismatched_types = {} annotations = {} for scls in reversed(cls.__mro__): if hasattr(scls, "__annotations__"): annotations.update(scls.__annotations__) for attr, type_ in annotations.items(): if hasattr(scls.__dict__, attr) and not isinstance( scls.__dict__[attr], type_ ): mismatched_types[attr] = scls.__dict__[attr] if mismatched_types: raise TypeError( f"Please provide expected type(s) for attrs:\n\t{mismatched_types}." "\n({class_arg: , ...})" ) return cls def __setattr__(cls, attr, val): if callable(val): val = enforce_method_types(val) else: _check_attr_matches_cls_annotation(cls, attr, val) super().__setattr__(attr, val) ``` * The `__prepare__` method returns a dictionary a `_RestrictedSetattrDict` instance. * The `__new__` method adds the custom `__setattr__` to the class namespace. It also checks statically defined class attributes against their annotations, if any. For the same attribute, the annotation of a subclass of `Typed` takes precedence over its superclass. A `TypeError` is raised if that's not the case. * `__setattr__` on the metaclass is what is called whenever a class attribute is added to the class after the class has been created by the metaclass (i.e. it can be triggered even in the `__new__` method of the metaclass). In other words, it is called when `cls.attribute = value` is executed. The full code is thus: ```python from inspect import signature from functools import wraps class _Untyped: pass class OverrideDisallowed(Exception): pass def _get_cls_attr_annotation(cls, attr): for scls in cls.__mro__: if attr in getattr(scls, "__annotations__", ()): return scls.__annotations__[attr] return _Untyped def _check_attr_matches_cls_annotation(cls, attr, val): expected_type = _get_cls_attr_annotation(cls, attr) if not (expected_type is _Untyped or isinstance(val, expected_type)): raise TypeError(f"Expected type {expected_type} for attribute {attr}") def get_func_annotations(signature): return { arg: param.annotation for arg, param in signature.parameters.items() if not param.annotation is signature.empty } def enforce_method_types(method): @wraps(method) def wrapper(*args, **kwargs): sig = signature(method) annotations = get_func_annotations(sig) bound_args = sig.bind(*args, **kwargs) mismatched_types = {} for arg, expected_type in annotations.items(): if arg in annotations and not isinstance( bound_args.arguments[arg], expected_type ): mismatched_types[arg] = expected_type if mismatched_types: raise TypeError( f"Please provide expected type(s) for:\n\t{mismatched_types}." "\n({arg: , ...})" ) # SILLY since we can only see the return type after running the method! return_val = method(*args, **kwargs) if (sig.return_annotation is not sig.empty) and not isinstance( return_val, sig.return_annotation ): raise TypeError(f"Expected return type {sig.return_annotation}.") return return_val return wrapper class _RestrictedSetattrDict(dict): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._setattr_assigned = False def __setitem__(self, key, val): # Prevent overrides of '__setattr__' added by the metaclass. if key == "__setattr__": if not self._setattr_assigned: self._setattr_assigned = True else: raise OverrideDisallowed("Cannot override __setattr__") if callable(val): val = enforce_method_types(val) super().__setitem__(key, val) class _TypedMeta(type): @classmethod def __prepare__(mcs, name, bases, **kwargs): return _RestrictedSetattrDict() def __new__(mcs, name, bases, namespace, **kwargs): # cls instance attribute setting. def set_attr(self, attr, val): expected_type = _get_cls_attr_annotation(type(self), attr) if not (expected_type is _Untyped or isinstance(val, expected_type)): raise TypeError(f"Expected type {expected_type} for attribute {attr}") object.__setattr__(self, attr, val) namespace.__setitem__("__setattr__", set_attr) cls = super().__new__(mcs, name, bases, namespace) # Check if class attributes match annotations, if set. # cls's annotations take precedence over subclasses, hence # 'reversed(cls.__mro__)'. mismatched_types = {} annotations = {} for scls in reversed(cls.__mro__): if hasattr(scls, "__annotations__"): annotations.update(scls.__annotations__) for attr, type_ in annotations.items(): if hasattr(scls.__dict__, attr) and not isinstance( scls.__dict__[attr], type_ ): mismatched_types[attr] = scls.__dict__[attr] if mismatched_types: raise TypeError( f"Please provide expected type(s) for attrs:\n\t{mismatched_types}." "\n({class_arg: , ...})" ) return cls def __setattr__(cls, attr, val): if callable(val): val = enforce_method_types(val) else: _check_attr_matches_cls_annotation(cls, attr, val) super().__setattr__(attr, val) class Typed(metaclass=_TypedMeta): ... ``` To illustrate the usage of the metaclass, we have: ```python # Example 1. class User(Typed): # Throws TypeError since d is set to an int. d: str = 1 # Example 2. class User(Typed): d: str def __init__(self, a: int, b, c: str): self.a = a self.b = b self.c = c self.d = 1 # TypeError. # Throws TypeError since `self.d` is set to an int. user = User(1, 2, 3) # Example 3. class User(Typed): d: str def __init__(self, a: int, b, c: str): self.a = a self.b = b self.c = c user = User(1, 2, "3") # Works. user.a = "2" # TypeError. user.b = "3" # Works. user.c = 1 # TypeError. # Example 4. # class User(Typed): d: str = "a" def foo(self, x: int): -> int return "1" # Class attributes. User.d = 2 # TypeError. User.d = "1" # Works. # Instance attributes. user = User() user.foo(1) # Return value TypeError. user.foo("1") # Argument TypeError. ``` ## Caveats and final words The above implementation gives odd results for `property` getters and setters, methods decorated with `@classmethod`, or descriptors in general. Function annotations for these aren't enforced. Methods decorated with `@staticmethod` behave the quirkiest, as illustrated below. Now let’s not begin to imagine the horrors of wrongly using a `Typed` class with multiple inheritance. ```python class User(Typed): @staticmethod def foo(x: int) -> int: return "1" User.foo("2") # Argument TypeError. User.foo(2) # Return value TypeError. user = User() user.foo() # TypeError. user.foo(2) # Too many arguments provided to `sig.bind` in `enforce_method_types`. ``` This kind of 'magic' without extensive documentation could lead to a debugging nightmare when working with classes subclassing `Typed`. Also, metaclasses might be an overkill for simpler situations. Class decorators can be used for simpler cases, but they have the drawback of being uninheritable. The `__init_subclass__` special method can be used to create inheritable class decorators. Metaclass usage is quite common in framework development. For example, it enables the automatic addition of several utility methods for Django models. To see an even more exploratory idea of creating a 'typed' python environment, check out David Beazley's Pycon talk on metaclasses: https://youtu.be/sPiWg5jSoZI?si=e\_bZwnlqhYiMe6ba.