Atomicity

Atomicity ensures that for a database (DB) operation with a set of queries, any evaluated query (a query that leads to the database being "touched") that raises a DatabaseError (or any of its subclasses like django.db.utils.IntegrityError) should lead to such set being terminated, and to the set's DB changes being rolled back.

If no such exceptional query exists in such a set, then all queries in the set are run and committed to the DB.

An operation comprising a set of queries on the DB with atomicity being enforced on such a set is called a transaction.

Since either all or none of the queries in a transaction are committed, you can't test that code internal to a transaction runs.

In Django, a block defining a set of operations for which its atomicity on the DB is guaranteed is called an atomic block and can be created with atomic() (django.transaction.atomic).

An operation in an atomic block is either committed or rolled back, depending on whether or not any exception is raised at any point in such a block.

Changes by non-query code in an atomic block would still be seen if such operations are placed before the previously described exception-raising code, and side-effects of such code would persist beyond the block's execution.

Example:

with transaction.atomic():  # How an atomic block is defined.
    print("Transaction begins!")  # Runs.
    update_db_entry_and_save()  # Assume no exception.
    raise_exception()  # Exception -> leave block.
    print("Aw, man!")  # Doesn't run.
    do_a_query()  # Doesn't run either.

# update_db_entry_and_save()'s changes are rolled back due to the
# raised exception (which could be from an integrity violating
# query).

Savepoints

Atomic blocks can be nested to create subtransactions (savepoints). A successful subtransaction's db changes could be rolled back if the outer atomic block raises an exception after the inner block's execution.

Example:

with transaction.atomic():
    with transaction.atomic():  # Savepoint.
        update_db_entry_and_save()  # Assume no exception.
    raise_exception()

# Due to raise_exception(), whatever DB changes are made
# in update_db_entry_and_save() are rolled back.

A savepoint can be thought of as a safe rollback point for when a subtransaction doesn't complete, thus avoiding a complete rollback of the full transaction.

atomic() takes a durable argument which when set to True ensures that an atomic block must be the outermost atomic block. If such a block is nested, a RuntimeError is raised.

Handling exceptions in an atomic block

The proper way to handle an exception raised in an atomic block is to wrap the entire atomic block in a try-except block, and not by handling the exception in the atomic block (except the handled exception is from a subtransaction handled in the same prescribed structure).

If an atomic block handles such an exception in the wrong way, it may hide the fact that something has gone wrong from the atomic() context manager and thus violate the atomicity of such block on the DB.

Example:

try:
    with transaction.atomic():
        non_query_code()  # Runs.
        successful_query()  # Runs.
        raise_exception()
except ...:
    do_whatever()  # Can execute queries, too.

# Here, successful_query()'s changes are rolled back due to
# raise_exception().

Example:

# Wrong way to handle exceptions in an atomic block.
with transaction.atomic():
    non_query_code()  # Runs
    successful_query_code1()  # Runs
    try:
        raise_exception()
    except ...:
        pass
    query_code2()  # Runs. (Assume it raises no exception.)

If DatabaseError (or its subclass) is raised manually and not due to a query, i.e. raise_exception() == raise DatabaseError (or its subclass), then all the queries in the above block (including raise_exception's DB queries that run successfully) get committed breaking atomicity.

If otherwise, then query_code2() raises django.db.transaction.TransactionManagementError since the transaction has been explicitly violated by a query in raise_exception(), and no other query should run in the atomic block.

raise_exception() raising other types of Exception (e.g. ValueError) also means that all queries in the above block get committed as the error has been explicitly handled without Django's knowledge, breaking atomicity.

Example:

# Nested atomic block (subtransaction) error handling.
with transaction.atomic():
    # Wrap the subtransaction in an try-except block.
    try:
        with transaction.atomic():
            update_db_entry_and_save()
            query_violates_integrity()
    except IntegrityError:
        pass
    print("Weeee!")  Runs.
    do_successful_query()  # Runs.
    do_non_query()  # Runs.

# In this structure, exception -> update_db_entry_and_save()'s
# changes are rolled back by the nested atomic block.

Relation to TestCase and TransactionTestCase

A TransactionTestCase resets the DB to a known state after each test (method) by truncating the table such that the next test gets a clean cursor. Hence, the whole test block runs without being atomic.

A TransactionTestCase's tests would thus run in autocommit mode (Django's default mode), where each query is automatically committed to the DB unless a transaction is active. You can test safely whether a transaction runs or not since queries raising IntegrityError don't lead to each test block being exited as each test is non-atomic.

A TransactionTestCase's tests may thus call commit() and rollback() (from django.db.transaction) to observe their effects on the DB. You could also manually set savepoints and disable autocommit.

A TestCase wraps both the test class and its tests in transactions to speed up the DB reset (no slow table truncations).

Calling commit(), rollback(), or trying to change the DB's autocommit state in a TestCase test will fail as the test block is an atomic block: attempting to commit or roll back changes, or change the autocommit state of the DB in an atomic block breaks atomicity.

Consider the following possible structures:

# Structure a. 
TransactionTestCase_class_non_atomic_block:
    test1_non_atomic_block:
        do_stuff_in_test1()

    test2_non_atomic_block:
        do_stuff_in_test2()
    ...

# Structure b.
TestCase_class_atomic_block:
    test1_atomic_block:
        do_stuff_in_test1()

    test2_atomic_block:
        do_stuff_in_test2()
    ...

Now consider a case where we use structure a. Suppose do_stuff_in_test1() follows this pattern:

try:
    do_stuff_on_db()
except IntegrityError:
    handle()
execute_another_query()

If do_stuff_on_db() raises an IntegrityError, execute_another_query() runs without problems since the test is a regular code block that doesn't enforce atomicity. (For instance, TransactionTestCase.assertRaises(IntegrityError, ...) could substitute for the try-except block.)

If, however, we use a TestCase and try such a thing, execute_another_query() raises TransactionManagementError as described previously.

A solution with TestCase

A way around this apparent limitation is to wrap a query that could raise an exception in a subtransaction, and then wrap the subtransaction in a try-except block. This way you can perform other queries in the test without errors, since the atomic block ensures that whatever IntegrityError raised by such query leads to the subtransaction's DB changes being rolled back, without compromising the atomicity for the whole test's atomic block.

The following would thus be a valid pattern for do_stuff_in_test1() in the above TestCase structure:

try:
    with transaction.atomic():
        do_stuff_on_db()
except IntegrityError:
    handle()

successful_query_code()  # Runs

try:
    with transaction.atomic():
        do_other_db_stuff()
except IntegrityError:
    handle()
...