Why Are My Tests Failing?

Oh No! My Tests are Failing!

How many times have you run into this situation:

You're working on a project and you've just finished writing a new feature. You're feeling pretty good about it, so you run your tests to make sure everything is working as expected. But then you see it - a failing test.

Or, even worse, multiple failing tests that are seemingly unrelated to the changes you just made.

This has happened to me so many times, and it can be really frustrating. But it's also a great opportunity to learn more about your codebase and improve your testing skills.

In this article, I'm going to share some tips for dealing with failing tests in your Laravel applications. I'll cover some common reasons why tests fail, how to diagnose the problem, and how to fix it.

Don't Blindly Update the Test

When you see a failing test, your first instinct might be to update the test to make it pass. But before you do that, take a step back and investigate why the test is failing.

Is the test highlighting a new error or bug in the code? Or is there something wrong with the test itself? Has the code changed since the test was last updated?

You don't want to blindly update the test to make it pass. If the test was in fact highlighting a bug in the code, updating the test could be covering up that bug. This could lead to bigger problems down the line.

For example, let's take this function that calculates the sum of two numbers:

        
1function sum(int $a, int $b): int
2{
3    return $a + $b;
4}    

And this test that checks if the function returns the correct sum:

        
1#[Test]
2public function numbers_are_added_correctly(): void
3{
4    $result = sum(3, 3);
5 
6    $this->assertSame(6, $result);
7}    

Now let's say, for some bizarre reason, the function is accidentally updated to multiply the numbers instead of adding them:

        
1function sum(int $a, int $b): int
2{
3    return $a * $b;
4}    

In this case, the test would be correctly highlighting an error. If you were to blindly update the test to expect the result to be 9, you would be covering up the bug in the code.

In this case, you might want to look at the Git history for the code related to the test and see if there have been any changes that could have caused the test to fail.

Are Your Tests Out of Date?

It may be possible that your tests are in fact out of date and need to be updated to reflect the latest changes in your codebase.

For example, let's imagine we want to update our sum function to accept and return floats instead of integers:

        
1function sum(float $a, float $b): float
2{
3    return $a + $b;
4}    

In this case, the existing test would now fail because we were using $this->assertSame(6, $result) which checks for strict equality. We would need to update the test to use $this->assertSame(6.0, $result) instead.

If you run into this situation, it's a perfectly valid reason to update the test, assuming the changes to the code are intentional.

Has Your Environment Changed?

Another reason your tests might be failing is that your environment has changed since the last time you ran them.

Common factors that can affect your tests include:

• Composer dependencies have been updated - Have you updated a package recently that could be causing the tests to fail?
• PHP version has changed - Are you running the tests on a different version of PHP than before?
• External dependencies have changed - Have you updated your database, cache, storage, or any other external services that your tests rely on?

Was There a Recent Merge Conflict?

We've all been there. We try and avoid touching the same code as our team members, but sometimes it's unavoidable and we end up with a merge conflict.

If you've recently resolved a merge conflict, it's possible that the code has been broken in the process. This could be a reason why your tests are failing.

So it might be worth checking the code that was affected by the merge conflict to see if there are any issues.

Are There Random Values in Your Factories?

One of the most puzzling reasons why tests might fail is due to random values in your factories. I have run into this problem more times than I'd like to admit.

Imagine you have a factory that generates an App\Models\Post model for your tests like so:

        
 1namespace Database\Factories;
 2 
 3use Illuminate\Database\Eloquent\Factories\Factory;
 4 
 5class PostFactory extends Factory
 6{
 7    public function definition(): array
 8    {
 9        return [
10            'title' => $this->faker->sentence(),
11            'slug' => $this->faker->slug(),
12            'description' => $this->faker->sentence(),
13            'content' => $this->faker->paragraph(),
14            'status' => $this->faker->randomElement([
15                'draft',
16                'published',
17                'in_review',
18            ]),
19        ];
20    }
21}    

As we can see, the status field is randomly set to one of three values: draft, published, or in_review.

Let's say we have a test that checks that only published posts can be viewed in our application:

        
1#[Test]
2public function published_post_can_be_viewed(): void
3{
4    $publishedPost = Post::factory()->create();
5 
6    $response = $this->get(route('posts', $publishedPost))
7        ->assertOk();
8}    

We've made a slight error in this test! We've not specified that we only want to create published posts.

Assuming that the status field has an equal chance of being set to published, draft, or in_review, there's a 66% chance that the test will fail because the post is not published.

But it might be that the test passes when you first write it because the factory randomly generates a published post. So it's only when you run the test again that it fails.

To fix this, we need to update the test to specify that we only want to create published posts:

        
1#[Test]
2public function published_post_can_be_viewed(): void
3{
4    $publishedPost = Post::factory()->create(['status' => 'published']);
5 
6    $response = $this->get(route('posts', $publishedPost))
7        ->assertOk();
8}    

Now, this test should always pass because we're explicitly creating a published post.

These types of errors can be very difficult to spot, especially when the chances of failing values are much lower or the test is not run frequently.

A great way to check whether this is what's causing the issue is to rerun the test many times over. For instance, you might want to run test the 10 or 100 times to see if any of the test runs fail.

How to Diagnose the Problem

If you need help to diagnose why your tests might be failing, here are a few things you can try:

Run the tests in a different place

If the tests are failing on your local machine, try running them on a different machine or in a CI environment. This can help you determine if the problem is specific to your environment.

Run the tests individually

If you have a large test suite, try running the failing tests individually to see if you can isolate the problem to a specific test or group of tests. Although your tests should be independent of each other, sometimes one test can affect the outcome of another if they've not been configured correctly.

Check the test output

Look at the output of the failing test to see if there are any error messages or stack traces that could give you a clue as to what's going wrong.

Check out older versions of the code

If you're using version control (which you should be), you can check out older versions of the code to see if the tests were passing at that point. This can help you identify when the problem was introduced.

I like doing this myself because it can keep checking out older versions of the code until I can get back to a point where the tests were passing. This is also really helpful for pinpointing when problems were introduced from updating external packages and libraries.

Use Other Tools

You can use other tools in your testing arsenal to help diagnose the problem. For example, in the context of PHP and Laravel, I might want to use something like Larastan to check for static analysis errors or Pest architecture testing to see if the problem is related to the structure of the project.

In our code examples above, Larastan would have been able to pick up the type mismatch in the sum function and let us know that the test needed updating.

So we could have used this as evidence to update the test, rather than blindly updating it.

Ask Someone Else

If you're really stuck, don't be afraid to ask for help. Sometimes a fresh pair of eyes can spot something you've missed.

Likewise, it might be that the person you're asking has run into the same problem before and can offer some advice on how to fix it.

It's much better to spend 5 minutes asking someone for help than to spend hours trying to fix the problem yourself.

Conclusion

Hopefully, this article has given you an insight into why your tests might be failing, and how to get them working again.

If you enjoyed reading this post, you might be interested in checking out my 220+ page ebook "Battle Ready Laravel" which covers similar topics in more depth.

Or, you might want to check out my other 440+ ebook "Consuming APIs in Laravel" which teaches you how to use Laravel to consume APIs from other services.

If you're interested in getting updated each time I publish a new post, feel free to sign up for my newsletter below.

Keep on building awesome stuff! 🚀