Add reason for failed match to RaisesGroup #3145

jakkdl · 2024-11-29T12:16:19Z

As another preparatory step for adding RaisesGroup into pytest, I thought I'd finally get around to making it.. not a nightmare to debug why it fails to match. When I've been using RaisesGroup with a complicated structure I've sometimes found it very tricky to figure out why it's not matching - and I imagine that's 10x worse for somebody not intimately familiar with its innards.

This does introduce a major change in behavior, previously RaisesGroup, like pytest.raises, would silently pass through the exception if it didn't match - but now it will instead fail with an AssertionError. This also has precedent upstream though, pytest.raises will fail with an AssertionError iff you've specified match.

You still see the original exception in the traceback, and in many ways I think always failing with an AssertionError is more legible.
I don't think this will impact end user's test suites in a majority of cases, unless they're either testing RaisesGroup behavior itself, or doing some very weird nesting. But even if so, they can rewrite their code as:

with pytest.raises(AssertionError) as exc_info:
    with RaisesGroup(...):
        ...
assert isinstance(exc_info.type, ValueError)

Another improvement would be making .matches() directly raise an AssertionError, instead of quietly setting .fail_reason. This would break any test currently doing if not RaisesGroup(...).matches(...):, or even more plausibly assert not RaisesGroup(...).matches(...), so
I think I should add a new method .assert_matches() to both RaisesGroup and Matcher, which either calls .matches() and asserts the return value with fail_reason as the message - or do it the other way around and have .matches() call .assert_matches() in a try:.

There's lots of bikeshedding possible with the phrasing of each error message, and am not completely happy with nested cases, so would very much appreciate any suggestions.

…age when it fails to match

codecov · 2024-11-29T12:18:12Z

Codecov Report

Attention: Patch coverage is 99.05956% with 3 lines in your changes missing coverage. Please review.

Project coverage is 99.98396%. Comparing base (b41bb13) to head (580137d).

Files with missing lines	Patch %	Lines
src/trio/testing/_raises_group.py	98.55072%	3 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                  @@
##                 main       #3145         +/-   ##
====================================================
- Coverage   100.00000%   99.98396%   -0.01605%     
====================================================
  Files             124         124                 
  Lines           18460       18700        +240     
  Branches         1216        1265         +49     
====================================================
+ Hits            18460       18697        +237     
- Misses              0           3          +3

Files with missing lines	Coverage Δ
src/trio/_tests/test_exports.py	`100.00000% <100.00000%> (ø)`
src/trio/_tests/test_testing_raisesgroup.py	`100.00000% <100.00000%> (ø)`
src/trio/testing/_raises_group.py	`99.10979% <98.55072%> (-0.89021%)`	⬇️

….. though somewhat skeptical of their complexity cost vs benefit

src/trio/_tests/test_exports.py

Zac-HD

I'm excited to see continued progress here! Really good support for ExceptionGroup in testing is one of those things that doesn't seem like a huge problem until you're living with it 😅

src/trio/_tests/test_testing_raisesgroup.py

Zac-HD · 2024-11-29T20:42:18Z

src/trio/_tests/test_testing_raisesgroup.py

+    with pytest.raises(
+        AssertionError,
+        match=wrap_escape(
+            "Raised exception did not match: RuntimeError():\n"
+            "  RuntimeError() is not of type 'ValueError'\n"
+            "  Matcher(TypeError): RuntimeError() is not of type 'TypeError'\n"
+            "  RaisesGroup(RuntimeError): RuntimeError() is not an exception group, but would match with `allow_unwrapped=True`\n"
+            "  RaisesGroup(ValueError): RuntimeError() is not an exception group",
+        ),
+    ):
+        with RaisesGroup(
+            ValueError,
+            Matcher(TypeError),
+            RaisesGroup(RuntimeError),
+            RaisesGroup(ValueError),
+        ):
+            raise ExceptionGroup(
+                "a",
+                [RuntimeError(), TypeError(), ValueError(), ValueError()],
+            )


Even after staring at this for a while, I'm having trouble matching up the corresponding lines to matchers to leaf exceptions. Can you add some comments laying out the intent?

I think that's mostly because RaisesGroup doesn't care about the order of matches, but I'm concerned that a greedy algorithm will fail to find a passing match when one exists (although exhaustive checks could be quite expensive in the worst case!).

I think our error messages are also unhelpful or even misleading here, in that the lines of output all seem to be describing the first leaf exception, without accounding for the fact that they might be (close) matches for other leaves or subgroups. Instead, I suggest trying to match each sub-exception.

Set aside whatever matchers and subgroups have a 1:1 correspondence. This is probably the common case; i.e. one or more missing or unexpected exceptions. Report each missing exception and unused matcher.

If it's more complicated than that - for example, two TypeErrors and only one matcher - report all of those as well. I don't think it's worth trying to work out which are "closer to matching", or which to set aside; the user can sort it out.

sorry yeah, it's failing to match any of the expected exceptions with the first exception in the group. But had to pass more exceptions in the group to make it pass the number-of-exceptions check. Will add comments :)

The algorithm is currently described in the class docstring:

trio/src/trio/testing/_raises_group.py

Lines 376 to 382 in 993a173

The matching algorithm is greedy, which means cases such as this may fail::

with RaisesGroups(ValueError, Matcher(ValueError, match="hello")):

raise ExceptionGroup("", (ValueError("hello"), ValueError("goodbye")))

even though it generally does not care about the order of the exceptions in the group.

To avoid the above you should specify the first ValueError with a Matcher as well.

But I suppose I should at the very least also mention it in the docstring of matches().

I very much like your idea, will give it a shot!
This probably also means we should stop abandoning as soon as the number of exceptions is wrong, with RaisesGroup(ValueError, TypeError): raise ExceptionGroup("", [ValueError]) can point out that TypeError is the one unmatched exception.
This makes my current suggest-flattening logic impossible though, so I'll go ahead with run-it-again-but-with-flattening-if-it-fails.

Made an attempt at generating a more verbose error message, and while this is an intentionally complicated example it's also fairly reasonable that a user expects a complicated structure, but instead gets a very different exceptiongroup with very different errors.

So for this code:

with RaisesGroup( RaisesGroup(ValueError), RaisesGroup(RaisesGroup(ValueError)), RaisesGroup(Matcher(TypeError, match="foo")), RaisesGroup(TypeError, ValueError) ): raise ExceptionGroup( "", [TypeError(), ExceptionGroup("", [TypeError()]), ExceptionGroup("", [TypeError(), TypeError()])] )

my current WIP implementation outputs the following:

=========================================================================================== FAILURES =========================================================================================== __________________________________________________________________________________ test_assert_message_nested __________________________________________________________________________________ + Exception Group Traceback (most recent call last): | File "/home/h/Git/trio/raisesgroup_fail_reason/src/trio/_tests/test_testing_raisesgroup.py", line 533, in test_assert_message_nested | raise ExceptionGroup( | ExceptionGroup: (3 sub-exceptions) +-+---------------- 1 ---------------- | TypeError +---------------- 2 ---------------- | ExceptionGroup: (1 sub-exception) +-+---------------- 1 ---------------- | TypeError +------------------------------------ +---------------- 3 ---------------- | ExceptionGroup: (2 sub-exceptions) +-+---------------- 1 ---------------- | TypeError +---------------- 2 ---------------- | TypeError +------------------------------------ During handling of the above exception, another exception occurred: def test_assert_message_nested() -> None: > with RaisesGroup( RaisesGroup(ValueError), RaisesGroup(RaisesGroup(ValueError)), RaisesGroup(Matcher(TypeError, match="foo")), RaisesGroup(TypeError, ValueError) ): E AssertionError: Raised exception did not match: Failed to match expected and raised exceptions. E The following expected exceptions did not find a match: {RaisesGroup(Matcher(TypeError, match='foo')), RaisesGroup(ValueError), RaisesGroup(RaisesGroup(ValueError)), RaisesGroup(TypeError, ValueError)} E The following raised exceptions did not find a match E TypeError(): E RaisesGroup(Matcher(TypeError, match='foo')): TypeError() is not an exception group E RaisesGroup(ValueError): TypeError() is not an exception group E RaisesGroup(RaisesGroup(ValueError)): TypeError() is not an exception group E RaisesGroup(TypeError, ValueError): TypeError() is not an exception group E ExceptionGroup('', [TypeError()]): E RaisesGroup(Matcher(TypeError, match='foo')): Failed to match: Matcher(TypeError, match='foo'): Regex pattern 'foo' did not match '' E RaisesGroup(ValueError): Failed to match: TypeError() is not of type 'ValueError' E RaisesGroup(RaisesGroup(ValueError)): Failed to match: RaisesGroup(ValueError): TypeError() is not an exception group E RaisesGroup(TypeError, ValueError): Too few exceptions raised! The following expected exception(s) found no match: {<class 'ValueError'>} E ExceptionGroup('', [TypeError(), TypeError()]): E RaisesGroup(Matcher(TypeError, match='foo')): Failed to match expected and raised exceptions. E The following expected exceptions did not find a match: {Matcher(TypeError, match='foo')} E The following raised exceptions did not find a match E TypeError(): E Matcher(TypeError, match='foo'): Regex pattern 'foo' did not match '' E TypeError(): E Matcher(TypeError, match='foo'): Regex pattern 'foo' did not match '' E E RaisesGroup(ValueError): Failed to match expected and raised exceptions. E The following expected exceptions did not find a match: {<class 'ValueError'>} E The following raised exceptions did not find a match E TypeError(): E TypeError() is not of type 'ValueError' E TypeError(): E TypeError() is not of type 'ValueError' E E RaisesGroup(RaisesGroup(ValueError)): Failed to match expected and raised exceptions. E The following expected exceptions did not find a match: {RaisesGroup(ValueError)} E The following raised exceptions did not find a match E TypeError(): E RaisesGroup(ValueError): TypeError() is not an exception group E TypeError(): E RaisesGroup(ValueError): TypeError() is not an exception group E E RaisesGroup(TypeError, ValueError): Failed to match expected and raised exceptions. E The following expected exceptions did not find a match: {<class 'ValueError'>} E The following raised exceptions did not find a match E TypeError(): E TypeError() is not of type 'ValueError' E It matches <class 'TypeError'> which was paired with TypeError() src/trio/_tests/test_testing_raisesgroup.py:527: AssertionError

There's about a million things to tweak, and this doesn't hit all logic (in particular when some-but-not-all exceptions match), but do you like this as the basic structure? It feels kind of insane to have an exception message that's 45 lines - and given the quadratic nature could quickly spiral even further - but maybe that's fine?

src/trio/_tests/test_testing_raisesgroup.py

Zac-HD · 2024-11-29T20:57:30Z

src/trio/_tests/test_testing_raisesgroup.py

+            # TODO: try to avoid printing the check function twice?
+            # it's very verbose with printing out memory location.
+            # And/or don't print memory location and just print the name
+            "Raised exception did not match: OSError(6, ''):\n"
+            f"  Matcher(OSError, check={check_errno_is_5!r}): check {check_errno_is_5!r} did not return True for OSError(6, '')",


Matcher(OSError, check={check_errno_is_5!r}): check did not return True for OSError(6, '') seems reasonable to me? (and see above for Hypothesis' pprinter idea...)

The problem with that is in the RaisesGroup case, if you don't want RaisesGroup(RaisesGroup(ValueError, check=my_fun)) to print the function twice - then RaisesGroup(ValueError, check=my_fun) will not print it at all.
The only way out of that is to not print it in the overview (maybe something like Raises(ValueError, check=...): check {check_errno_is_5!r} did not return True for OsError(6, '')), or to use logic to see if we're nested or not (I suppose we already have _depth)*

Never printing it in the un-nested case is probably fine a majority of the time, but if you're doing something like for f in ...: with RaisesGroup(..., check=f) then you'd need an extra debugging cycle.

* I'll go ahead with this latter one for now

fixed that case, but still printing it twice in nested cases:

trio/src/trio/_tests/test_testing_raisesgroup.py

Lines 718 to 726 in a8e263c

# in nested cases you still get it multiple times though...

with pytest.raises(

AssertionError,

match=wrap_escape(

f"Raised exception group did not match: RaisesGroup(Matcher(OSError, check={check_errno_is_5!r})): Matcher(OSError, check={check_errno_is_5!r}): check did not return True for OSError(6, '')",

),

):

with RaisesGroup(RaisesGroup(Matcher(OSError, check=check_errno_is_5))):

raise ExceptionGroup("", [ExceptionGroup("", [OSError(6, "")])])

src/trio/_tests/test_testing_raisesgroup.py

src/trio/testing/_raises_group.py

…n some cases. Simplify message in trivial cases.

Zac-HD · 2024-12-10T23:05:09Z

src/trio/_tests/test_testing_raisesgroup.py

+    with pytest.raises(
+        AssertionError,
+        match=wrap_escape(
+            "Raised exception group did not match: SyntaxError() is not of type 'ValueError'",


we do this dance often enough that a helper function might be in order,

def fails_raises_group(msg): prefix = "Raised exception group did not match: " return pytest.raises(AssertionError, match=wrap_escape(prefix + msg))

and I'd also consider the parenthesized-with statement now that we've dropped Python 3.8

I'm somewhat skeptical of the parenthesized-with here because we're testing the context manager, rather than the innermost body being what the focus is on, but testing it just for the sake of exciting "new" feature ^^

Zac-HD · 2024-12-11T00:02:14Z

src/trio/_tests/test_testing_raisesgroup.py

+            "  ExceptionGroup('', [ValueError(), TypeError()]):\n"
+            "    ExceptionGroup('', [ValueError(), TypeError()]) is not of type 'ValueError'\n"
+            "    ExceptionGroup('', [ValueError(), TypeError()]) is not of type 'TypeError'\n"


I'm fine with printing a table here, but we could just remove the repetition:

Suggested change

" ExceptionGroup('', [ValueError(), TypeError()]):\n"

" ExceptionGroup('', [ValueError(), TypeError()]) is not of type 'ValueError'\n"

" ExceptionGroup('', [ValueError(), TypeError()]) is not of type 'TypeError'\n"

" ExceptionGroup('', [ValueError(), TypeError()]):\n"

" -> not of type 'ValueError'\n"

" -> not of type 'TypeError'\n"

(particularly because I expect most reprs to be considerably longer than this)

This turned out to be very messy, so I instead ended up printing type(e).__name__ on a type mismatch, and not print the repr if check failed, and I think that ultimately resolved it cleanly. There's some non-nested simple cases that now doesn't print the repr at all, but I think they should all be fine.

src/trio/_tests/test_testing_raisesgroup.py

Zac-HD · 2024-12-11T00:10:10Z

src/trio/_tests/test_testing_raisesgroup.py

+            "  ValueError('bar'):\n"
+            "    RaisesGroup(RuntimeError): ValueError('bar') is not an exception group\n"
+            "    RaisesGroup(ValueError): ValueError('bar') is not an exception group, but would match with `allow_unwrapped=True`\n"
+            "    It matches <class 'ValueError'> which was paired with ValueError('foo')",


this note is going to save someone hours of frustration 👍

I think this is also the reason to get rid of

trio/src/trio/testing/_raises_group.py

Lines 765 to 768 in 8281d7c

# TODO: I'm not 100% sure about printing the exact same as in the case above

# And in all cases of len(failed_attemts_dict)==1 we probably don't need a full table

if 1 == len(remaining_exceptions) == len(failed_attempts_dict):

return f"{succesful_str}{failed_attempts_dict.popitem()[1][0]}"

When RaisesGroup is sensitive to ordering I think it can be quite important to print duplicate matches. I was thinking "if there's only exception that didn't get matched, and only one expected that had no raised, then we surely only care why those two didn't match" but while that may be the majority of cases it's not always the case.

(edited because I mixed up my scenarios)

edit2: wait no, I can just add another condition to see if there's any "would've-paired-but-ordering-made-them-not". But that's for another day

… on check fail. Fix output issues when multiple copies of the same object were expected/raised

…. Also fix succesful->successful typo.

…raised-exception by adding another example

…and suggest using Matcher if match would match against a lone expected & raised exception

jakkdl · 2025-01-08T16:32:24Z

The only remaining TODO now is

trio/src/trio/_tests/test_testing_raisesgroup.py

Lines 103 to 110 in 9612d1e

    
           # TODO: this one should maybe say that it matched an already matched exception 
        
           with ( 
        
               fails_raises_group( 
        
                   "1 matched exception. Too few exceptions raised, found no match for: [<class 'ValueError'>]" 
        
               ), 
        
               RaisesGroup(ValueError, ValueError), 
        
           ): 
        
               raise ExceptionGroup("", (ValueError(),))

which I'm procrastinating on because _check_exceptions is becoming an unwieldy beast.
I think I'm sticking to my guns on the matching being greedy, but _check_exceptions is so close to being fully exhaustive on fail already that we can probably have a message for "there is a pairing that would've worked, you should specify your requirements more stringently to allow the greedy matching to work"

CoolCat467 · 2025-01-09T00:34:43Z

Run failure for pypi looks familiar, I think the issue last time was that package cache pypi runner was using had cached an outdated version of what uv versions were available, leading it to believe that constraint-specified version of uv does not exist when it actually does in reality, just using an old cache.

…h no matches. Print if there's a possible covering that failed due to too-general expected.

jakkdl · 2025-01-09T17:30:15Z

Code is perhaps a bit messy, and the tests definitely are (I think I'll restructure them when moving to pytest), but otherwise I'm happy with current functionality. So please review away!

Zac-HD

I suggest we check the two tweaks to messages below, and then come back for the hypothesis-plugin-monkeypatch in a followup PR. Otherwise, looks great and I can't wait to use it!

Zac-HD · 2025-01-10T01:21:29Z

src/trio/testing/_raises_group.py

+
+
+def _check_repr(check: Callable[[BaseExcT_1], bool]) -> str:
+    """Split out so it can be monkeypatched (e.g. by hypothesis)"""


Suggested change

"""Split out so it can be monkeypatched (e.g. by hypothesis)"""

"""Split out so it can be monkeypatched (e.g. by our hypothesis plugin)"""

and then in this function, add

try: from ..testing import raises_group from hypothesis.internal.reflection import get_pretty_function_description raises_group._check_repr = get_pretty_function_description except ImportError: pass

Zac-HD · 2025-01-10T01:36:07Z

src/trio/testing/_raises_group.py

+        assert res is not NotChecked
+        # why doesn't mypy pick up on the above assert?
+        return res  # type: ignore[return-value]


I think because type[NotChecked] allows subclasses; if you did assert not issubclass(res, NotChecked) it might work?

Zac-HD · 2025-01-10T02:00:42Z

src/trio/_tests/test_testing_raisesgroup.py

+    # and `isinstance(raised, expected_type)`.
+    with (
+        fails_raises_group(
+            "Regex pattern 'foo' did not match 'bar' of 'ExceptionGroup', but matched expected exception. You might want RaisesGroup(Matcher(ValueError, match='foo'"


maybe "matched the inner exception"?

Zac-HD · 2025-01-10T03:55:58Z

src/trio/_tests/test_testing_raisesgroup.py

+            # Ideally the "did you mean to re.escape" should be indented twice
+            "Matcher(match='h(ell)o'): Regex pattern 'h(ell)o' did not match 'h(ell)o'\n"
+            "Did you mean to `re.escape()` the regex?",


Could we just put " " at the start of the message to get this effect?

Zac-HD · 2025-01-10T03:58:11Z

src/trio/_tests/test_testing_raisesgroup.py

+    # we only get one instance of aaaaaaaaaa... and bbbbbb..., but we do get multiple instances of ccccc... and dddddd..
+    # but I think this now only prints the full repr when that is necessary to disambiguate exceptions
+    with (
+        fails_raises_group(


This is indeed pretty verbose, but I think that's still the right tradeoff to pick for being explicit 👍

Zac-HD · 2025-01-10T04:00:41Z

src/trio/_tests/test_testing_raisesgroup.py

+            match=(
+                r"^Raised exception group did not match: \n"
+                r"The following expected exceptions did not find a match:\n"
+                r"  Matcher\(check=<function test_assert_message_nested.<locals>.<lambda> at .*>\)\n"


ah... if we do the monkeypatch from Trio's hypothesis plugin as I suggest above, we'll then also want to use the maybe-monkeypatched _check_repr function to construct the expected string; and maybe also assert that that string is one of two known possibilities. Which is kinda ugly, but imo showing the lambda source here is nice enough to be worth it.

Zac-HD · 2025-01-10T04:25:39Z

src/trio/testing/_raises_group.py

+    def get_all_results_for_actual(self, actual: int) -> Iterator[str | None]:
+        for res in self.results[actual]:
+            assert res is not NotChecked
+            yield res  # type: ignore[misc]


According to codecov, this function is never executed. Can we delete it?

A5rocks

I haven't looked that hard at the logic but this looks good! (note to self: restart review at _check_exceptions)

This is mostly wording/nitpicks/minor changes. A bunch of them, but it's a few concerns duplicated:

I don't like _depth and proposed some changes to get rid of it
__str__ defaults to __repr__
honestly I can't remember, probably just suggestions about errors/docs

A5rocks · 2025-01-10T05:35:39Z

src/trio/_tests/test_exports.py

+        # Ignore classes that don't use attrs, they only define their members once
+        # __init__ is called (and reason they don't use attrs is because they're going
+        # to be reimplemented in pytest).
+        # I think that's why these are failing at least?


You could try tossing attrs on them and seeing! But I think so too.

Maybe even just try putting __slots__ on them and using that to double check, then removing this (checked) comment.

A5rocks · 2025-01-10T05:37:20Z

newsfragments/3145.feature.rst

@@ -0,0 +1 @@
+`trio.testing.RaisesGroup` now raises an `AssertionError` with a helpful message if it fails to match the raised exception. Previously it would let the original exception fall through, but that is now instead the ``__context__``.


Suggested change

`trio.testing.RaisesGroup` now raises an `AssertionError` with a helpful message if it fails to match the raised exception. Previously it would let the original exception fall through, but that is now instead the ``__context__``.

`trio.testing.RaisesGroup` now raises an `AssertionError` with a helpful message if it fails to match the raised exception. Previously it would let the original exception fall through, but the raised exception is now instead the ``__context__`` of `AssertionError`.

I think it might be useful to say how to get back the raised exception here. Also maybe some sort of rewording to put "if it fails" earlier to avoid people thinking this effects them. But both of those are... unnecessary.

A5rocks · 2025-01-10T05:39:38Z

src/trio/testing/_raises_group.py

@@ -8,6 +8,7 @@
    Generic,
    Literal,
    cast,
+    final as t_final,


Why not use the final imported below? It prevents subclassing too.

A5rocks · 2025-01-10T05:42:11Z

src/trio/testing/_raises_group.py

        if isinstance(match, str):
-            self.match = re.compile(match)
+            self.match = re.compile(match, flags=_REGEX_NO_FLAGS)


I'm confused: is this not the default? I see _REGEX_NO_FLAGS = re.compile(r"").flags.

A5rocks · 2025-01-10T05:46:05Z

src/trio/testing/_raises_group.py

-            self.exception_type,
-        ):
+        self.fail_reason = _check_type(self.exception_type, exception)
+        if self.fail_reason is not None:


I think you should comment above the section of defining these _check_x that they're seperated for monkey patching purposes. I assume so right? Otherwise this could be simpler:

Suggested change

if self.fail_reason is not None:

if self._check_type(exception):

Where self._check_type sets self.fail_reason and returns a boolean.

A5rocks · 2025-01-10T06:16:39Z

src/trio/testing/_raises_group.py

+                assert (
+                    self.match_expr is not None
+                ), "can't be None if _check_match failed"
+                self.fail_reason += f", but matched expected exception. You might want RaisesGroup(Matcher({expected.__name__}, match={_match_pattern(self.match_expr)!r}"


Suggested change

self.fail_reason += f", but matched expected exception. You might want RaisesGroup(Matcher({expected.__name__}, match={_match_pattern(self.match_expr)!r}"

self.fail_reason += f", but matched expected exception. You might want RaisesGroup(Matcher({expected.__name__}, match={_match_pattern(self.match_expr)!r}))"

A5rocks · 2025-01-10T06:21:11Z

src/trio/testing/_raises_group.py

+                res = _check_expected(
+                    expected, actual_exceptions[i_rem], _depth=_depth + 3
+                )


Use textwrap.indent instead.

A5rocks · 2025-01-10T06:21:28Z

src/trio/testing/_raises_group.py

+                results.set_result(
+                    i_exp,
+                    i_actual,
+                    _check_expected(expected, actual, _depth=_depth + 3),


Use textwrap.indent instead.

A5rocks · 2025-01-10T06:23:05Z

src/trio/testing/_raises_group.py

+    def _check_exceptions(
+        self,
+        actual_exceptions: Sequence[BaseException],
+        _depth: int,


I don't think this is necessary. I also don't like how it needs to be threaded everywhere. I've left further comments on where we recurse with a larger depth about indenting the result instead.

A5rocks · 2025-01-10T06:24:56Z

src/trio/testing/_raises_group.py

+        self.fail_reason = self._check_exceptions(
+            actual_exceptions,
+            _depth=_depth,
+        )
+
+        if self.fail_reason is not None:


Suggested change

self.fail_reason = self._check_exceptions(

actual_exceptions,

_depth=_depth,

)

if self.fail_reason is not None:

if not self._check_exceptions(

actual_exceptions,

_depth=_depth,

):

Just mutate self.fail_reason in self._check_exceptions. This gets rid of the # type: ignore later.

jakkdl added 2 commits November 27, 2024 17:49

RaisesGroup will now raise an assertionerror with an informative mess…

03a3959

…age when it fails to match

suggest allow_unwrapped if it would match with it

c0dcb58

jakkdl requested a review from Zac-HD November 29, 2024 12:16

jakkdl added 4 commits November 29, 2024 13:27

fix test fails

2b98b2f

codecov and fix suggestion to re.escape string

f84d9d1

add _check_expected helper

df27595

remove trailing period. Add trying_flattening and suggest_flattening.…

993a173

….. though somewhat skeptical of their complexity cost vs benefit

CoolCat467 reviewed Nov 29, 2024

View reviewed changes

src/trio/_tests/test_exports.py Show resolved Hide resolved

Zac-HD reviewed Nov 29, 2024

View reviewed changes

jakkdl added 3 commits December 3, 2024 16:40

simplify & remove false positive for suggestion to use flatten_subgroups

656e040

overhaul message in the complicated case. Avoid printing check repr i…

debe557

…n some cases. Simplify message in trivial cases.

Merge remote-tracking branch 'origin/main' into raisesgroup_fail_reason

a8e263c

jakkdl requested a review from Zac-HD December 9, 2024 16:11

Zac-HD reviewed Dec 11, 2024

View reviewed changes

jakkdl and others added 12 commits December 18, 2024 13:26

Merge branch 'main' into raisesgroup_fail_reason

8281d7c

Merge remote-tracking branch 'origin/main' into raisesgroup_fail_reason

3f757f1

add fails_raises_group test helper

a62a252

parenthesized-with

5efafd5

add newsfragment

6ab9ac9

print type(e).__name__ instead of repr on type fail. Don't print repr…

f38d63b

… on check fail. Fix output issues when multiple copies of the same object were expected/raised

don't print brief error if there's a possibility of misordering

b6ce586

Merge remote-tracking branch 'origin/main' into raisesgroup_fail_reason

ebbf694

change message when encountering group and expecting a bare exception…

9748cc8

…. Also fix succesful->successful typo.

procrastinating on fixing expected-exception-matches-already-matched-…

89bf77e

…raised-exception by adding another example

specify 'of [base]exceptiongroup when regex fails against the group; …

ba938bc

…and suggest using Matcher if match would match against a lone expected & raised exception

fix sphinx

9612d1e

overhaul saving of results. Print would-have-matched for expected wit…

580137d

…h no matches. Print if there's a possible covering that failed due to too-general expected.

jakkdl marked this pull request as ready for review January 9, 2025 17:29

Zac-HD approved these changes Jan 10, 2025

View reviewed changes

Zac-HD reviewed Jan 10, 2025

View reviewed changes

A5rocks requested changes Jan 10, 2025

View reviewed changes

Zac-HD mentioned this pull request Jan 10, 2025

Using Generic Exceptions (ExceptionGroups in particular) with raises in a type-safe manner is not ideal pytest-dev/pytest#13115

Open

	The matching algorithm is greedy, which means cases such as this may fail::

	with RaisesGroups(ValueError, Matcher(ValueError, match="hello")):
	raise ExceptionGroup("", (ValueError("hello"), ValueError("goodbye")))

	even though it generally does not care about the order of the exceptions in the group.
	To avoid the above you should specify the first ValueError with a Matcher as well.

	# in nested cases you still get it multiple times though...
	with pytest.raises(
	AssertionError,
	match=wrap_escape(
	f"Raised exception group did not match: RaisesGroup(Matcher(OSError, check={check_errno_is_5!r})): Matcher(OSError, check={check_errno_is_5!r}): check did not return True for OSError(6, '')",
	),
	):
	with RaisesGroup(RaisesGroup(Matcher(OSError, check=check_errno_is_5))):
	raise ExceptionGroup("", [ExceptionGroup("", [OSError(6, "")])])

	# TODO: I'm not 100% sure about printing the exact same as in the case above
	# And in all cases of len(failed_attemts_dict)==1 we probably don't need a full table
	if 1 == len(remaining_exceptions) == len(failed_attempts_dict):
	return f"{succesful_str}{failed_attempts_dict.popitem()[1][0]}"



		def _check_repr(check: Callable[[BaseExcT_1], bool]) -> str:
		"""Split out so it can be monkeypatched (e.g. by hypothesis)"""

		@@ -0,0 +1 @@
		`trio.testing.RaisesGroup` now raises an `AssertionError` with a helpful message if it fails to match the raised exception. Previously it would let the original exception fall through, but that is now instead the ``__context__``.

	if self.fail_reason is not None:
	if self._check_type(exception):

	self.fail_reason += f", but matched expected exception. You might want RaisesGroup(Matcher({expected.__name__}, match={_match_pattern(self.match_expr)!r}"
	self.fail_reason += f", but matched expected exception. You might want RaisesGroup(Matcher({expected.__name__}, match={_match_pattern(self.match_expr)!r}))"

Add reason for failed match to RaisesGroup #3145

Are you sure you want to change the base?

Add reason for failed match to RaisesGroup #3145

Conversation

jakkdl commented Nov 29, 2024

codecov bot commented Nov 29, 2024 • edited Loading

Codecov Report

Zac-HD left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakkdl Jan 2, 2025 • edited Loading

Choose a reason for hiding this comment

jakkdl commented Jan 8, 2025

CoolCat467 commented Jan 9, 2025

jakkdl commented Jan 9, 2025 • edited Loading

Zac-HD left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

A5rocks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 29, 2024 •

edited

Loading

jakkdl Jan 2, 2025 •

edited

Loading

jakkdl commented Jan 9, 2025 •

edited

Loading