Introducing try..except*

[1st1]

The design discussed in this issue has been consolidated in https://github.com/python/exceptiongroups/blob/master/except_star.md.

---

Disclaimer

• I'm going to be using the ExceptionGroup name in this issue, even though there are other alternatives, e.g. AggregateException. Naming of the "exception group" object is outside of the scope of this issue.
• This issue is primarily focused on discussing the new syntax modification proposal for the try..except construct, shortly called "except*".
• I use the term "naked" exception for regular Python exceptions not wrapped in an ExceptionGroup. E.g. a regular ValueError propagating through the stack is "naked".
• I assume that ExceptionGroup would be an iterable object. E.g. list(ExceptionGroup(ValueError('a'), TypeError('b'))) would be equal to [ValueError('a'), TypeError('b')]
• I assume that ExceptionGroup won't be an indexable object; essentially it's similar to Python set. The motivation for this is that exceptions can occur in random order, and letting users write group[0] to access the "first" error is error prone. The actual implementation of ExceptionGroup will likely use an ordered list of errors though.
• I assume that ExceptionGroup will be a subclass of BaseException, which means it's assignable to Exception.__context__ and can be directly handled with except ExceptionGroup.
• The behavior of good and old regular try..except will not be modified.

Syntax

We're considering to introduce a new variant of the try..except syntax to simplify working with exception groups:

try:
  ...
except *SpamError:
  ...
except *BazError as e:
  ...
except *(BarError, FooError) as e:
  ...

The new syntax can be viewed as a variant of the tuple unpacking syntax. The * symbol indicates that zero or more exceptions can be "caught" and processed by one except * clause.

We also propose to enable "unpacking" in the raise statement:

errors = (ValueError('hello'), TypeError('world'))
raise *errors

Semantics

Overview

The except *SpamError block will be run if the try code raised an ExceptionGroup with one or more instances of SpamError. It would also be triggered if a naked instance of SpamError was raised.

The except *BazError as e block would aggregate all instances of BazError into a list, wrap that list into an ExceptionGroup instance, and assign the resultant object to e. The type of e would be ExceptionGroup[BazError]. If there was just one naked instance of BazError, it would be wrapped into a list and assigned to e.

The except *(BarError, FooError) as e would aggregate all instances of BarError or FooError into a list and assign that wrapped list to e. The type of e would be ExceptionGroup[Union[BarError, FooError]].

Even though every except* star can be called only once, any number of them can be run during handling of an ExceptionGroup. E.g. in the above example, both except *SpamError: and except *(BarError, FooError) as e: could get executed during handling of one ExceptionGroup object, or all of the except* clauses, or just one of them.

It is not allowed to use both regular except clauses and the new except* clauses in the same try block. E.g. the following example would raise a SyntaxErorr:

try:
   ...
except ValueError:
   pass
except *CancelledError:
   pass

Exceptions are mached using a subclass check. For example:

try:
  low_level_os_operation()
except *OSerror as errors:
  for e in errors:
    print(type(e).__name__)

could output:

BlockingIOError
ConnectionRefusedError
OSError
InterruptedError
BlockingIOError

New raise* Syntax

The new raise * syntax allows to users to only process some exceptions out of the matched set, e.g.:

try:
  low_level_os_operation()
except *OSerror as errors:
  new_errors = []
  for e in errors:
    if e.errno != errno.EPIPE:
       new_errors.append(e)
  raise *new_errors

The above code ignores all EPIPE OS errors, while letting all others propagate.

raise * syntax is special: it effectively extends the exception group with a list of errors without creating a new ExceptionGroup instance:

try:
  raise *(ValueError('a'), TypeError('b'))
except *ValueError:
  raise *(KeyError('x'), KeyError('y'))

# would result in: 
#   ExceptionGroup({KeyError('x'), KeyError('y'), TypeError('b')})

A regular raise would behave similarly:

try:
  raise *(ValueError('a'), TypeError('b'))
except *ValueError:
  raise KeyError('x')

# would result in: 
#   ExceptionGroup({KeyError('x'), TypeError('b')})

raise * accepts arguments of type Iterable[BaseException].

Unmatched Exceptions

Example:

try:
  raise *(ValueError('a'), TypeError('b'), TypeError('c'), KeyError('e'))
except *ValueError as e:
  print(f'got some ValueErrors: {e}')
except *TypeError as e:
  print(f'got some TypeErrors: {e}')
  raise *e

The above code would print:

got some ValueErrors: ExceptionGroup({ValueError('a')})
got some TypeErrors: ExceptionGroup({TypeError('b'), TypeError('c')})

And then crash with an unhandled KeyError('e') error.

Basically, before interpreting except * clauses, the interpreter will have an exception group object with a list of exceptions in it. Every except * clause, evaluated from top to bottom, can filter some of the exceptions out of the group and process them. In the end, if the exception group has no exceptions left in it, it wold mean that all exceptions were processed. If the exception group has some unprocessed exceptions, the current frame will be "pushed" to the group's traceback and the group would be propagated up the stack.

Exception Chaining

If an error occur during processing a set of exceptions in a except * block, all matched errors would be put in a new ExceptionGroup which would have its __context__ attribute set to the just occurred exception:

try:
  raise *(ValueError('a'), ValueError('b'), TypeError('z'))
except *ValueError:
  1 / 0

# would result in:
#
#   ExceptionGroup({
#     TypeError('z'),
#     ZeroDivisionError()
#   })
#
# where the `ZeroDivizionError()` instance would have
# its __context__ attribute set to
#
#   ExceptionGroup({
#     ValueError('a'), ValueError('b')
#   })

It's also possible to explicitly chain exceptions:

try:
  raise *(ValueError('a'), ValueError('b'), TypeError('z'))
except *ValueError as errors:
  raise RuntimeError('unexpected values') from errors

# would result in:
#
#   ExceptionGroup(
#     TypeError('z'),
#     RuntimeError('unexpected values')
#   )
#
# where the `RuntimeError()` instance would have
# its __cause__ attribute set to
#
#   ExceptionGroup({
#     ValueError('a'), ValueError('b')
#   })

See Also

• An analysis of how exception groups will likely be used in asyncio programs: Introducing try..catch #3 (comment)
• A WIP implementation of the ExceptionGroup type by @iritkatriel tracked here: GitHub - iritkatriel/cpython at exceptionGroup

[gvanrossum]

Very nice! Only a few nits.

Why do we need raise *errors? What's wrong with raise ExceptionGroup(*errors)? I think that should have the same semantics.

I think your English description is less precise than my example in #3 (comment). For example you don't specify in which order multiple handlers are run if more than one is run (mine makes it clear that they run from top to bottom) and I also think we should be careful to clarify what happens if you write e.g.

except OSError:
    ...
except BlockingIOError:  # Never runs
    ...

(To get the desired effect, the more specific handler should come first.)

Another thing: when an ExceptionGroup wraps another ExceptionGroup, iterating over the outer group should recurse into the inner group (i.e. flatten the structure); the matching process should do this too.

except *TypeError as e:
  print(f'got some TypeErrors: {e}')
  raise *e

Shouldn't this use bare raise?

Also probably should link to python-trio/trio#611 (Trio MultiError v2).

[iritkatriel]

Awesome!

If there was just one naked instance of BazError, it would be wrapped into a list and assigned to e.

This could sound like e ends up being a list. Maybe just say "it would be wrapped by an ExceptionGroup" (that inside it's in a list is an implementation detail). The following paragraph can be probably simplified a bit in the same way.

If an error occur during processing a set of exceptions in a except * block, all matched errors would be put in a new ExceptionGroup which would have its __context__ attribute set to the just occurred exception:

It's the other way around (as in your example) - the exception that just happened has context which is the ExceptionGroup that we are now handling.

Basically, before interpreting except * clauses, ....

It might be worth reiterating here what happens to exceptions raised within the except clauses (they are added to a separate list and merged with the unhandled exceptions at the end so they cannot be handled by another except clause of this try.)

• I assume that ExceptionGroup will be a subclass of BaseException

We considered (but I don't think we've decided yet) to make it a new type which is a superclass of BaseException, so that except BaseException doesn't catch it.

Still open: How control flow statements (return, continue, break) interact with a try-*except clause. Return should take us to finally. For continue/break I can think of more than one option.

[1st1]

@gvanrossum:

Why do we need raise *errors? What's wrong with raise ExceptionGroup(*errors)? I think that should have the same semantics.

Actually yes, I now think so too. Let me shed some light on why I added this syntax.

• In the initial version of this proposal, the type of e in except *ValueError as e: would be list[ValueError]. I wanted to hide ExceptionGroup from the user as much as possible. Then I realized that we also need to make __cause__ work, and that implies making it possible to write raise e from None etc. So I had to fix list[ValueError] to ExceptionGroup[ValueError].

• Second, quoting the proposal:

  raise * syntax is special: it effectively extends the exception group with a list of errors without creating a new ExceptionGroup instance:

  try:
    raise *(ValueError('a'), TypeError('b'))
  except *ValueError:
    raise *(KeyError('x'), KeyError('y'))
  
  # would result in: 
  #   ExceptionGroup({KeyError('x'), KeyError('y'), TypeError('b')})

  Basically, I wanted raise *errors to merge all errors back into the ExceptionGroup we're handling now. I now don't think it's a good idea: it's hard to explain to users and the distinction between the behavior of raise *errors and raise ExceptionGroup(*errors) would be utterly confusing.

So yeah, let's remove raise * until we find good motivation to include something like it. I'll update the proposal.

I think your English description is less precise than my example in #3 (comment). For example you don't specify in which order multiple handlers are run if more than one is run (mine makes it clear that they run from top to bottom) and I also think we should be careful to clarify what happens if you write e.g.

except OSError:
    ...
except BlockingIOError:  # Never runs
    ...

Right, I'll add a variant of this example to the proposal.

Another thing: when an ExceptionGroup wraps another ExceptionGroup, iterating over the outer group should recurse into the inner group (i.e. flatten the structure); the matching process should do this too.

Now this is something I'm really not sure about. Oh, I was just about to make an argument against this, when it struck me that you're right.

If there's a KeyboardInterrupt exception buried in some deeply nested ExceptionGroup you'd still want to catch it. I'll clarify this all in the proposal.

Shouldn't this use bare raise?

Yes. This is a remnant of my list[ValueError] idea; I'll fix this.

---

@iritkatriel

This could sound like e ends up being a list. Maybe just say "it would be wrapped by an ExceptionGroup" (that inside it's in a list is an implementation detail). The following paragraph can be probably simplified a bit in the same way.

Good catch! If you see my reply to @gvanrossum you'll find why ;) I'll fix this.

If an error occur during processing a set of exceptions in a except * block, all matched errors would be put in a new ExceptionGroup which would have its context attribute set to the just occurred exception:

It's the other way around (as in your example) - the exception that just happened has context which is the ExceptionGroup that we are now handling.

Another great catch, will fix.

It might be worth reiterating here what happens to exceptions raised within the except clauses (they are added to a separate list and merged with the unhandled exceptions at the end so they cannot be handled by another except clause of this try.)

Alright, I'll clarify.

We considered (but I don't think we've decided yet) to make it a new type which is a superclass of BaseException, so that except BaseException doesn't catch it.

Since they ExceptionGroup objects need to be able to be referenced from __cause__ and __context__ I think it's inevitable that they must be part of the exceptions class hierarchy. IMO, at the very least we'll need to introduce another BaseBaseException to it to avoid ExceptionGroup extending BaseException.

Still open: How control flow statements (return, continue, break) interact with a try-*except clause. Return should take us to finally. For continue/break I can think of more than one option.

I'll add examples.

[gvanrossum]

IMO, at the very least we'll need to introduce another BaseBaseException to it to avoid ExceptionGroup extending BaseException.

I would prefer not to do that. I will be backwards incompatible, since there is likely tons of code out there that either catches BaseException or uses isinstance(x, BaseException). Such code assumes that it'll get all exceptions; if it gets a BaseBaseException that isn't a BaseException their assumptions will be violated.

I know @njsmith said he would like ExceptionGroup to be something completely different from exceptions, but I think that's going too far. In the end it is an exception, is treated by an exception by all of the runtime, and you should be able to catch it with except BaseException (no *).

User code may have no business interacting with ExceptionGroup, but people will have to write infrastructure code in Python that does have to interact with it (e.g. custom loggers) so it should just be exposed as another BaseException with a custom API.

[1st1]

I would prefer not to do that. I will be backwards incompatible, since there is likely tons of code out there that either catches BaseException or uses isinstance(x, BaseException). Such code assumes that it'll get all exceptions; if it gets a BaseBaseException that isn't a BaseException their assumptions will be violated.

I know @njsmith said he would like ExceptionGroup to be something completely different from exceptions, but I think that's going too far. In the end it is an exception, is treated by an exception by all of the runtime, and you should be able to catch it with except BaseException (no *).

I'm also not entirely sure why we'd need that. If we make it completely different from exceptions then ExceptionGroup would still need to be duck-type compatible with BaseException on both Python & C level just so that everything works when it is assigned to __context__ and __cause__ of regular exceptions.

And I also don't see a good enough argument to deepen the Python exceptions hierarchy with BaseBaseException. IMO, making ExceptionGroup a BaseException is good enough. Moreover, logging code like

try:
  boostrap_app()
except BaseException as e:
  logger.log(e)
  raise

is better to continue working when exception groups are around.

[gvanrossum]

Still open: How control flow statements (return, continue, break) interact with a try-*except clause. Return should take us to finally. For continue/break I can think of more than one option.

How bad would it be if we just included these in the translated code, assuming the translated code is something like

except BaseException as _err:
    if not isinstance(_err, ExceptionGroup):
        _err = ExceptionGroup(_err)
    if (e := _err.split(E1)) is not None:
        <handler>  # break, continue, return just included here

Noting that there actually must be another try statement to deal with exceptions coming out of <handler> (they must be added to the separate list which will be merged with _err at the end), but that shouldn't affect break/continue/return. (This should probably be done using a try..finally around all the handlers together.)

However, another way to think of these is that they raise a special kind of pseudo-exception that isn't catchable but that, when it is re-raised at the end of the finally block, magically executes the desired control flow operation. This is how these are currently handled (though I don't recognize the code in ceval enough to be able to point to it without more research). Then however we have a problem if both except blocks get run in e.g.

except *E1:
    break
except *E2:
    continue

Which will it do? The first semantics I propose will clearly do break because at that point the handler for E2 will not even be called. But is that what we expect?

[iritkatriel]

Does the first option mean that break/continue acts on a loop enclosing this try-except block? So either break or continue means that no further except clauses are executed (but finally of course is)?

[1st1]

How about we prohibit continue and break inside except*? We can enable them in later versions of Python. But right now, it seems it's very hard to define them in a non-confusing way.

[aeros]

How about we prohibit continue and break inside except*? We can enable them in later versions of Python. But right now, it seems it's very hard to define them in a non-confusing way.

Agreed. I think that for now, continue and break add a significant degree of complexity that hasn't been defined to be practically useful enough within exception groups (at least not as of writing this, legitimate use cases may emerge as it matures). Even with those disabled, I think we're already hitting a threshold where the complexity is going to be a lot to digest for a single PEP.

[gvanrossum]

If we forbid break/continue we should also disallow return, which has the same issues. But while break/continue in except blocks feel pretty academic, return does not.

[1st1]

I decided that it would be much more productive to use GitHub PRs to start tracking changes to the proposal. I've created a document combining my motivation for the try..except* syntax as well as the opening of this issue in one file. And here's the first PR addressing feedback from @iritkatriel and @gvanrossum. Maybe we'll gradually morph that file into the PEP, we'll see.

[1st1]

The PR: #5