Prisma-style ORMs

Prisma, a popular ORM for TypeScript, allows writing queries like (adapted from this example):

const user = await prisma.user.findMany({
  select: {
    name: true,
    email: true,
    posts: true,
  },
});

for which the inferred type will be something like:

{
    email: string;
    name: string | null;
    posts: {
        id: number;
        title: string;
        content: string | null;
        authorId: number | null;
    }[];
}[]

Here, the output type is a combination of both existing information about the type of prisma.user and the type of the argument to findMany. It returns an array of objects containing the properties of user that were requested; one of the requested elements, posts, is a “relation” referencing another model; it has all of its properties fetched but not its relations.

We would like to be able to do something similar in Python, perhaps with a schema defined like:

class Comment:
    id: Property[int]
    name: Property[str]
    poster: Link[User]


class Post:
    id: Property[int]

    title: Property[str]
    content: Property[str]

    comments: MultiLink[Comment]
    author: Link[Comment]


class User:
    id: Property[int]

    name: Property[str]
    email: Property[str]
    posts: Link[Post]

(In Prisma, a code generator generates type definitions based on a prisma schema in its own custom format; you could imagine something similar here, or that the definitions were hand written)

and a call like:

db.select(
    User,
    name=True,
    email=True,
    posts=True,
)

which would have return type list[<User>] where:

class <User>:
    name: str
    email: str
    posts: list[<Post>]

class <Post>
    id: int
    title: str
    content: str

(Example code for implementing this below.)

Automatically deriving FastAPI CRUD models

In the FastAPI tutorial, they show how to build CRUD endpoints for a simple Hero type. At its heart is a series of class definitions used both to define the database interface and to perform validation/filtering of the data in the endpoint:

class HeroBase(SQLModel):
    name: str = Field(index=True)
    age: int | None = Field(default=None, index=True)


class Hero(HeroBase, table=True):
    id: int | None = Field(default=None, primary_key=True)
    secret_name: str


class HeroPublic(HeroBase):
    id: int


class HeroCreate(HeroBase):
    secret_name: str


class HeroUpdate(HeroBase):
    name: str | None = None
    age: int | None = None
    secret_name: str | None = None

The HeroPublic type is used as the return types of the read endpoint (and is validated while being output, including having extra fields stripped), while HeroCreate and HeroUpdate serve as input types (automatically converted from JSON and validated based on the types, using Pydantic).

Despite all multiple types and duplication here, mechanical rules could be written for deriving these types:

• Public should include all non-“hidden” fields, and the primary key should be made non-optional
• Create should include all fields except the primary key
• Update should include all fields except the primary key, but they should all be made optional and given a default value

With the definition of appropriate helpers, this proposal would allow writing:

class Hero(NewSQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)

    name: str = Field(index=True)
    age: int | None = Field(default=None, index=True)

    secret_name: str = Field(hidden=True)

type HeroPublic = Public[Hero]
type HeroCreate = Create[Hero]
type HeroUpdate = Update[Hero]

Those types, evaluated, would look something like:

class HeroPublic:
    id: int
    name: str
    age: int | None


class HeroCreate:
    name: str
    age: int | None = None
    secret_name: str


class HeroUpdate:
    name: str | None = None
    age: int | None = None
    secret_name: str | None = None

While the implementation of Public, Create, and Update are certainly more complex than duplicating code would be, they perform quite mechanical operations and could be included in the framework library.

A notable feature of this use case is that it depends on performing runtime evaluation of the type annotations. FastAPI uses the Pydantic models to validate and convert to/from JSON for both input and output from endpoints.

Currently it is possible to do the runtime half of this: we could write functions that generate Pydantic models at runtime based on whatever rules we wished. But this is unsatisfying, because we would not be able to properly statically typecheck the functions.

(Example code for implementing this below.)

dataclasses-style method generation

We would additionally like to be able to generate method signatures based on the attributes of an object. The most well-known example of this is probably generating __init__ methods for dataclasses, which we present a simplified example of. (In our test suites, this is merged with the FastAPI-style example above, but it need not be).

This kind of pattern is widespread enough that PEP 681 was created to represent a lowest-common denominator subset of what existing libraries do.

Make it possible for libraries to implement more of these patterns directly in the type system will give better typing without needing futher special casing, typechecker plugins, hardcoded support, etc.

(Example code for implementing this below.)

More powerful decorator typing

The typing of decorator functions has long been a pain point in python typing. The situation was substantially improved by the introducing of ParamSpec in PEP 612, but a number of patterns remain unsupported:

• Adding/removing/modifying a keyword parameter
• Modifying a variable number of parameters – XXX: check how well TypeVarTuple does this

This proposal will cover those cases.

XXX: Ehhhhhh the generic situation could be bad for some of it? For partial certainly, I think, which otherwise we can almost do.

NumPy-style broadcasting

One of the motivations for the introduction of TypeVarTuple in PEP 646 is to represent the shapes of multi-dimensional arrays, such as:

x: Array[float, L[480], L[640]] = Array()

The example in the that PEP shows how TypeVarTuple can be used to make sure that both sides of an arithmetic operation having matching shapes. Most multi-dimensional array libraries, however, also support broadcasting [1], which allows the mixing of differently shaped data. With this PEP, we can define a Broadcast[A, B] type alias, and then use it as a return type:

class Array[DType, *Shape]:
    def __add__[*Shape2](
        self,
        other: Array[DType, *Shape2]
    ) -> Array[DType, *Broadcast[tuple[*Shape], tuple[*Shape2]]]:
        raise BaseException

(The somewhat clunky syntax of wrapping the TypeVarTuple in another tuple is because typecheckers currently disallow having two TypeVarTuple arguments. A possible improvement would be to allow writing the bare (non-starred or Unpack-ed) variable name to mean its interpretation as a tuple.)

We can then do:

a1: Array[float, L[4], L[1]]
a2: Array[float, L[3]]
a1 + a2  # Array[builtins.float, Literal[4], Literal[3]]

b1: Array[float, int, int]
b2: Array[float, int]
b1 + b2  # Array[builtins.float, int, int]

err1: Array[float, L[4], L[2]]
err2: Array[float, L[3]]
# err1 + err2  # E: Broadcast mismatch: Literal[2], Literal[3]

TODO: Link the implementation

Unpack of typevars for **kwargs

A minor proposal that could be split out maybe:

Supporting Unpack of typevars for **kwargs:

def f[K: BaseTypedDict](**kwargs: Unpack[K]) -> K:
    return kwargs

Here BaseTypedDict is defined as:

class BaseTypedDict(typing.TypedDict):
    pass

But any typeddict would be allowed there. (Or, maybe we should allow dict?)

This is basically a combination of “PEP 692 – Using TypedDict for more precise **kwargs typing” and the behavior of Unpack for *args from “PEP 646 – Variadic Generics”.

This is potentially moderately useful on its own but is being done to support processing **kwargs with type level computation.

—

Extended Callables, take 2

We introduce a Param type the contains all the information about a function param:

class Param[N: str | None, T, Q: ParamQuals = typing.Never]:
    pass

ParamQuals = typing.Literal["*", "**", "default", "keyword"]

type PosParam[N: str | None, T] = Param[N, T, Literal["positional"]]
type PosDefaultParam[N: str | None, T] = Param[N, T, Literal["positional", "default"]]
type DefaultParam[N: str, T] = Param[N, T, Literal["default"]]
type NamedParam[N: str, T] = Param[N, T, Literal["keyword"]]
type NamedDefaultParam[N: str, T] = Param[N, T, Literal["keyword", "default"]]
type ArgsParam[T] = Param[Literal[None], T, Literal["*"]]
type KwargsParam[T] = Param[Literal[None], T, Literal["**"]]

And then, we can represent the type of a function like:

def func(
    a: int,
    /,
    b: int,
    c: int = 0,
    *args: int,
    d: int,
    e: int = 0,
    **kwargs: int
) -> int:
    ...

as (we are omiting the Literal in places):

Callable[
    [
        Param["a", int, "positional"],
        Param["b", int],
        Param["c", int, "default"],
        Param[None, int, "*"],
        Param["d", int, "keyword"],
        Param["e", int, Literal["default", "keyword"]],
        Param[None, int, "**"],
    ],
    int,
]

or, using the type abbreviations we provide:

Callable[
    [
        PosParam["a", int],
        Param["b", int],
        DefaultParam["c", int,
        ArgsParam[int, "*"],
        NamedParam["d", int],
        NamedDefaultParam["e", int],
        KwargsParam[int],
    ],
    int,
]

(Rationale discussed below.)

Grammar specification of the extensions to the type language

Note first that no changes to the Python grammar are being proposed, only to the grammar of what Python expressions are considered as valid types.

(It’s also slightly imprecise to call this a grammar: <bool-operator> refers to any of the names defined in the Boolean Operators section, which might be imported qualified or with some other name)

<type> = ...
     # Type booleans are all valid types too
     | <type-bool>

     # Conditional types
     | <type> if <type-bool> else <type>

     # Types with variadic arguments can have
     # *[... for t in ...] arguments
     | <ident>[<variadic-type-arg> +]

# Type conditional checks are boolean compositions of
# boolean type operators
<type-bool> =
      <bool-operator>[<type> +]
    | not <type-bool>
    | <type-bool> and <type-bool>
    | <type-bool> or <type-bool>
    | any(<type-bool-for>)
    | all(<type-bool-for>)

<variadic-type-arg> =
      <type> ,
    | * [ <type-for-iter> ] ,


<type-for> = <type> <type-for-iter>+ <type-for-if>*
<type-for-iter> =
      # Iterate over a tuple type
      for <var> in Iter[<type>]
<type-for-if> =
      if <type-bool>

(<type-bool-for> is identical to <type-for> except that the result type is a <type-bool> instead of a <type>.)

There are three core syntactic features introduced: type booleans, conditional types and unpacked comprehension types.

Type operators

In some sections below we write things like Literal[int]] to mean “a literal that is of type int”. I don’t think I’m really proposing to add that as a notion, but we could.

class InitField[KwargDict: BaseTypedDict]:
    def __init__(self, **kwargs: typing.Unpack[KwargDict]) -> None:
        ...

    def _get_kwargs(self) -> KwargDict:
        ...

class A:
    foo: int = InitField(default=0)

Lifting over Unions

Many of the builtin operations are “lifted” over Union.

For example:

Concat[Literal['a'] | Literal['b'], Literal['c'] | Literal['d']] = (
    Literal['ac'] | Literal['ad'] | Literal['bc'] | Literal['bd']
)

When an operation is lifted over union types, we take the cross product of the union elements for each argument position, evaluate the operator for each tuple in the cross product, and then union all of the results together. In Python, the logic looks like:

args_union_els = [get_union_elems(arg) for arg in args]
results = [
    eval_operator(*xs)
    for xs in itertools.product(*args_union_els)
]
if results:
    return Union[*results]
else:
    return Never

Runtime evaluation support

An important goal is supporting runtime evaluation of these computed types. We do not propose to add an official evaluator to the standard library, but intend to release a third-party evaluator library.

While most of the extensions to the type system are “inert” type operator applications, the syntax also includes list iteration and conditionals, which will be automatically evaluated when the __annotate__ method of a class, alias, or function is called.

In order to allow an evaluator library to trigger type evaluation in those cases, we add a new hook to typing:

• special_form_evaluator: This is a ContextVar that holds a callable that will be invoked with a typing._GenericAlias argument when __bool__ is called on a Boolean Operator or __iter__ is called on typing.Iter. The returned value will then have bool or iter called upon it before being returned.

  If set to None (the default), the boolean operators will return False while Iter will evaluate to iter(typing.TypeVarTuple("_IterDummy")). (TODO: Or should it be to iter([])?)

Prisma-style ORMs

First, to support the annotations we saw above, we have a collection of dummy classes with generic types.

class Pointer[T]:
    pass

class Property[T](Pointer[T]):
    pass

class Link[T](Pointer[T]):
    pass

class SingleLink[T](Link[T]):
    pass

class MultiLink[T](Link[T]):
    pass

The select method is where we start seeing new things.

The **kwargs: Unpack[K] is part of this proposal, and allows inferring a TypedDict from keyword args.

Attrs[K] extracts Member types corresponding to every type-annotated attribute of K, while calling NewProtocol with Member arguments constructs a new structural type.

GetName is a getter operator that fetches the name of a Member as a literal type–all of these mechanisms lean very heavily on literal types. GetMemberType gets the type of an attribute from a class.

def select[ModelT, K: typing.BaseTypedDict](
    typ: type[ModelT],
    /,
    **kwargs: Unpack[K],
) -> list[
    typing.NewProtocol[
        *[
            typing.Member[
                typing.GetName[c],
                ConvertField[typing.GetMemberType[ModelT, typing.GetName[c]]],
            ]
            for c in typing.Iter[typing.Attrs[K]]
        ]
    ]
]: ...

ConvertField is our first type helper, and it is a conditional type alias, which decides between two types based on a (limited) subtype-ish check.

In ConvertField, we wish to drop the Property or Link annotation and produce the underlying type, as well as, for links, producing a new target type containing only properties and wrapping MultiLink in a list.

type ConvertField[T] = (
    AdjustLink[PropsOnly[PointerArg[T]], T]
    if typing.IsSub[T, Link]
    else PointerArg[T]
)

PointerArg gets the type argument to Pointer or a subclass.

GetArg[T, Base, I] is one of the core primitives; it fetches the index I type argument to Base from a type T, if T inherits from Base.

(The subtleties of this will be discussed later; in this case, it just grabs the argument to a Pointer).

type PointerArg[T: Pointer] = typing.GetArg[T, Pointer, Literal[0]]

AdjustLink sticks a list around MultiLink, using features we’ve discussed already.

type AdjustLink[Tgt, LinkTy] = (
    list[Tgt] if typing.IsSub[LinkTy, MultiLink] else Tgt
)

And the final helper, PropsOnly[T], generates a new type that contains all the Property attributes of T.

type PropsOnly[T] = list[
    typing.NewProtocol[
        *[
            typing.Member[typing.GetName[p], PointerArg[typing.GetType[p]]]
            for p in typing.Iter[typing.Attrs[T]]
            if typing.IsSub[typing.GetType[p], Property]
        ]
    ]
]

The full test is in our test suite.

Automatically deriving FastAPI CRUD models

We have a more fully-worked example in our test suite, but here is a possible implementation of just Public

# Extract the default type from an Init field.
# If it is a Field, then we try pulling out the "default" field,
# otherwise we return the type itself.
type GetDefault[Init] = (
    GetFieldItem[Init, Literal["default"]]
    if typing.IsSub[Init, Field]
    else Init
)

# Create takes everything but the primary key and preserves defaults
type Create[T] = typing.NewProtocol[
    *[
        typing.Member[
            typing.GetName[p],
            typing.GetType[p],
            typing.GetQuals[p],
            GetDefault[typing.GetInit[p]],
        ]
        for p in typing.Iter[typing.Attrs[T]]
        if not typing.IsSub[
            Literal[True],
            GetFieldItem[typing.GetInit[p], Literal["primary_key"]],
        ]
    ]
]

The Create type alias creates a new type (via NewProtocol) by iterating over the attributes of the original type. It has access to names, types, qualifiers, and the literal types of initializers (in part through new facilities to handle the extremely common = Field(...) like pattern used here.

Here, we filter out attributes that have primary_key=True in their Field as well as extracting default arguments (which may be either from a default argument to a field or specified directly as an initializer).

dataclasses-style method generation

# Generate the Member field for __init__ for a class
type InitFnType[T] = typing.Member[
    Literal["__init__"],
    Callable[
        [
            typing.Param[Literal["self"], Self],
            *[
                typing.Param[
                    typing.GetName[p],
                    typing.GetType[p],
                    # All arguments are keyword-only
                    # It takes a default if a default is specified in the class
                    Literal["keyword"]
                    if typing.IsSub[
                        GetDefault[typing.GetInit[p]],
                        Never,
                    ]
                    else Literal["keyword", "default"],
                ]
                for p in typing.Iter[typing.Attrs[T]]
            ],
        ],
        None,
    ],
    Literal["ClassVar"],
]
type AddInit[T] = typing.NewProtocol[
    InitFnType[T],
    *[x for x in typing.Iter[typing.Members[T]]],
]

Extended Callables

We need extended callable support, in order to inspect and produce callables via type-level computation. mypy supports extended callables but they are deprecated in favor of callback protocols.

Unfortunately callback protocols don’t work well for type level computation. (They probably could be made to work, but it would require a separate facility for creating and introspecting methods, which wouldn’t be any simpler.)

I am proposing a fully new extended callable syntax because:

1. The mypy_extensions functions are full no-ops, and we need real runtime objects
2. They use parentheses and not brackets, which really goes against the philosophy here.
3. We can make an API that more nicely matches what we are going to do for inspecting members (We could introduce extended callables that closely mimic the mypy_extensions version though, if something new is a non starter)

Renounce all cares of runtime evaluation

This would have a lot of simplifying features.

TODO: Expand

Support TypeScript style pattern matching in subtype checking

This would almost certainly only be possible if we also decide not to care about runtime evaluation, as above.

Use type operators for conditional and iteration

Instead of writing:

• tt if tb else tf
• *[tres for T in Iter[ttuple]]

we could use type operator forms like:

• Cond[tb, tt, tf]
• UnpackMap[ttuple, lambda T: tres]
• or UnpackMap[ttuple, T, tres] where T must be a declared TypeVar

Boolean operations would likewise become operators (Not, And, etc).

The advantage of this is that constructing a type annotation never needs to do non-trivial computation, and thus we don’t need runtime hooks to support evaluating them.

It would also mean that it would be much easier to extract the raw type annotation. (The lambda form would still be somewhat fiddly. The non-lambda form would be trivial to extract, but requiring the declaration of a TypeVar goes against the grain of recent changes.)

Another advantage is not needing any notion of a special <type-bool> class of types.

The disadvantage is that is that the syntax seems a lot worse. Supporting filtering while mapping would make it even more bad (maybe an extra argument for a filter?).

We can explore other options too if needed.

Make the type-level operations more “strictly-typed”

This proposal is less “strictly-typed” than typescript (strictly-kinded, maybe?).

Typescript has better typechecking at the alias definition site: For P[K], K needs to have keyof P…

We could do potentially better but it would require more meachinery.

• KeyOf[T] - literal keys of T
• Member[T], when statically checking a type alias, could be treated as having some type like tuple[Member[KeyOf[T], object, str, ..., ...], ...]
• GetMemberType[T, S: KeyOf[T]] - but this isn’t supported yet. TS supports it.
• We would also need to do context sensitive type bound inference

What exactly are the subtyping (etc) rules for unevaluated types

Because of generic functions, there will be plenty of cases where we can’t evaluate a type operator (because it’s applied to an unresolved type variable), and exactly what the type evaluation rules should be in those cases is somewhat unclear.

Currently, in the proof of concept implementation in mypy, stuck type evaluations implement subtype checking fully invariantly: we check that the operators match and that every operand matches in both arguments invariantly.