Enabling Reruns For GH Actions With Tofu Updates Addressing Stale Terraform Plans

September 30, 2025 by StackCamp Team 82 views

Hey everyone! Let's dive into a common challenge we face when using GitHub Actions for infrastructure updates, specifically with Terraform (tofu) plans. We're going to break down an issue where rerunning a failed action can lead to a stale plan error, explore why this happens, and discuss potential solutions to ensure our deployments are smooth and reliable. So, grab your favorite beverage, and let's get started!

Understanding the Stale Plan Problem

So, here's the scenario, terraform plan is the core of this problem. You kick off a Terraform plan within your GitHub Actions workflow. Something goes wrong – maybe a misconfiguration, a network hiccup, or a temporary API outage. The plan fails. No biggie, you fix the issue and try to rerun the action, but you encounter this frustrating error message: Error: Saved plan is stale. What gives?

This error pops up because Terraform's state file has been modified since the original plan was created. Think of the Terraform state as a snapshot of your infrastructure. When you create a plan, Terraform compares your desired state (defined in your configuration files) with the current state (stored in the state file) and figures out the necessary changes. If another operation modifies the state in the meantime – perhaps another workflow run or a manual change – the original plan becomes outdated or "stale." It's like trying to use an old map for a newly constructed city; the roads just don't line up anymore.

Why is this a problem? Well, it prevents us from easily recovering from failures. We fix the initial issue, but we're still stuck because the plan can't be applied. This can lead to delays, frustration, and potentially even more significant problems if we can't deploy critical infrastructure updates promptly. This situation highlights the need for a robust mechanism to handle plan reruns and ensure state consistency in our CI/CD pipelines. It's crucial to remember that the Terraform state is the single source of truth for your infrastructure, and any inconsistencies can lead to unpredictable and potentially damaging outcomes.

Diving Deeper: Why Does State Change?

To truly grasp the issue, let's explore the common reasons why Terraform state might change between the plan and apply stages. One frequent culprit is concurrent operations. If multiple workflows or individuals are making changes to the same infrastructure concurrently, the state file can be modified between the plan and apply steps of a single workflow. This is especially common in larger teams or organizations where multiple people might be working on the same infrastructure.

Another reason is manual intervention. Sometimes, engineers might make manual changes to the infrastructure outside of the automated workflow. This could be for troubleshooting, emergency fixes, or simply to test something quickly. While manual interventions can be necessary, they can also easily lead to state drift if not properly documented and integrated back into the Terraform configuration. Imagine someone manually creating a resource in the cloud console; the Terraform state file will no longer reflect the actual state of the infrastructure.

External factors can also play a role. For instance, some cloud providers might automatically modify resources behind the scenes, leading to state changes. These changes might be due to security updates, maintenance operations, or other internal processes. While these changes are usually transparent, they can still impact the Terraform state.

Finally, unexpected errors during the apply stage can also leave the state in an inconsistent state. If a resource fails to create or update properly, Terraform might not be able to accurately record the state, leading to discrepancies. In such cases, it's crucial to carefully inspect the state file and potentially perform manual reconciliation steps.

Ensuring State Consistency

Okay, so we understand the problem. Now, how do we tackle it? The core of the solution lies in ensuring that the Terraform state remains consistent between the plan and apply stages. There are several strategies we can employ to achieve this. One of the most effective is to rerun the Terraform plan in the deployment step. This means that instead of relying on a previously saved plan, we generate a fresh plan right before applying the changes. This ensures that the plan is based on the most up-to-date state, minimizing the risk of staleness.

This approach adds a bit of overhead, as we're running the plan twice, but the benefits in terms of reliability and consistency are well worth it. It's like double-checking your directions before starting a road trip; it might take a few extra minutes, but it can save you from getting lost.

Another crucial aspect is state locking. Terraform supports state locking, which prevents concurrent operations from modifying the state file simultaneously. This is essential in preventing conflicts and ensuring data integrity. When a workflow acquires a state lock, other workflows attempting to modify the state will be blocked until the lock is released. This mechanism is like a traffic light for your infrastructure changes, ensuring that only one change happens at a time.

We also need to think about idempotency. Idempotency means that applying the same operation multiple times has the same effect as applying it once. In the context of Terraform, this means that if a resource fails to create or update and we retry the operation, it should succeed without causing further issues. Ensuring that our Terraform configurations are idempotent is crucial for handling failures and retries gracefully.

Practical Steps for Implementing Solutions

Let's translate these concepts into actionable steps. First, modify your GitHub Actions workflow to rerun the Terraform plan in the deployment step. This typically involves adding a step that executes terraform plan immediately before the terraform apply step. Make sure this step uses the same configuration and variables as the original plan step to ensure consistency.

Next, configure state locking. Terraform supports various backends for storing state, such as AWS S3, Azure Blob Storage, and Google Cloud Storage. Most of these backends support state locking out of the box. You'll need to configure your Terraform backend to enable state locking. This usually involves adding a few lines of configuration to your Terraform settings.

Consider using Terraform Cloud or HashiCorp Cloud Platform (HCP). These platforms provide advanced features for state management, collaboration, and automation. They also offer built-in state locking and concurrency control mechanisms, simplifying the process of managing your infrastructure state. These platforms are like having a dedicated control tower for your infrastructure, providing visibility and control over your deployments.

Finally, implement proper error handling and retry mechanisms in your workflows. If a Terraform command fails, your workflow should be able to detect the failure and retry the operation. This can be achieved using conditional steps and loops in your GitHub Actions workflow. Think of this as having a backup plan in case things don't go as expected.

Rerunning TF Plan in the Deployment Step: A Deep Dive

Let's zoom in on the idea of rerunning the Terraform plan in the deployment step. This might seem redundant at first – why plan twice? – but it's a powerful technique for ensuring consistency. The key advantage is that the second plan is generated using the most recent state file. This eliminates the risk of applying a stale plan, even if the state has been modified since the initial plan.

Here's how it works in practice. Your workflow might have two main stages: a planning stage and a deployment stage. In the planning stage, you run terraform plan to generate a plan file. This plan file is then typically saved as an artifact. In the deployment stage, instead of directly applying the saved plan, you rerun terraform plan to generate a fresh plan. This new plan is then applied using terraform apply. This seemingly small change makes a huge difference in the robustness of your deployments.

Why is this so effective? Imagine a scenario where a colleague manually modifies a resource after the initial plan is generated. If you were to apply the saved plan, you'd be overwriting their changes, potentially leading to data loss or other issues. By rerunning the plan, you're incorporating their changes into the new plan, ensuring that your deployment is aligned with the current state of the infrastructure. It's like having a real-time view of your infrastructure, ensuring that your changes are always in sync.

Implementing the Two-Plan Approach in GitHub Actions

To implement this approach in GitHub Actions, you'll need to modify your workflow file. Here's a basic example of how you might structure your workflow:

name: Terraform Deployment

on:
  push:
    branches:
      - main

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.0.0
      - name: Terraform Init
        run: terraform init
      - name: Terraform Plan
        run: terraform plan -out=tfplan
      - name: Upload Plan
        uses: actions/upload-artifact@v3
        with:
          name: tfplan
          path: tfplan

  apply:
    runs-on: ubuntu-latest
    needs: plan
    steps:
      - uses: actions/checkout@v3
      - uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.0.0
      - name: Download Plan
        uses: actions/download-artifact@v3
        with:
          name: tfplan
      - name: Terraform Init
        run: terraform init
      - name: Terraform Plan (Refresh)
        run: terraform plan -out=tfplan # Re-run the plan here
      - name: Terraform Apply
        run: terraform apply tfplan

In this example, the apply job downloads the plan artifact from the plan job, but then it reruns terraform plan before applying the changes. This ensures that the plan is based on the latest state.

Important Considerations: When implementing this approach, there are a few things to keep in mind. First, ensure that your Terraform configurations are idempotent. This means that applying the same plan multiple times should have the same effect as applying it once. This is crucial for preventing unexpected changes or errors. Second, consider adding drift detection mechanisms to your workflow. Drift detection involves periodically comparing the actual state of your infrastructure with the desired state defined in your Terraform configurations. This can help you identify and address any manual changes or inconsistencies that might have occurred.

Addressing Specific Concerns: Hotosm and k8s-infra

Now, let's bring this back to the specific context of hotosm and k8s-infra. The original issue highlighted a scenario where a Terraform plan failed, and subsequent reruns were blocked due to a stale plan error. This is a common challenge in collaborative environments where multiple individuals or teams might be working on the same infrastructure.

In this context, the strategies we've discussed – rerunning the plan in the deployment step, state locking, and idempotency – are particularly relevant. By implementing these measures, we can significantly reduce the risk of stale plan errors and ensure more reliable deployments. Additionally, consider using Terraform Cloud or HCP to simplify state management and collaboration.

Specific recommendations for hotosm and k8s-infra might include: Establishing clear guidelines for manual interventions, implementing a robust review process for Terraform configurations, and leveraging Terraform Cloud or HCP for state management and collaboration. It's also important to educate team members on the importance of state consistency and the potential risks of stale plans.

Moving Forward: Best Practices for Terraform in GitHub Actions

To wrap things up, let's outline some best practices for using Terraform in GitHub Actions to ensure smooth and reliable deployments. These practices build upon the concepts we've discussed and provide a holistic approach to managing your infrastructure as code.

Rerun the Terraform plan in the deployment step: This is the cornerstone of ensuring state consistency and preventing stale plan errors.
Configure state locking: Prevent concurrent operations from modifying the state file simultaneously.
Ensure idempotency: Make sure your Terraform configurations are idempotent to handle retries and failures gracefully.
Use a remote backend for state storage: Store your Terraform state in a remote backend like AWS S3, Azure Blob Storage, or Google Cloud Storage. This provides durability, consistency, and support for state locking.
Implement drift detection: Periodically compare the actual state of your infrastructure with the desired state to identify and address inconsistencies.
Use Terraform Cloud or HCP: Consider using these platforms for advanced features like state management, collaboration, and automation.
Establish clear guidelines for manual interventions: Define a process for making manual changes to the infrastructure and ensure that these changes are properly documented and integrated back into the Terraform configuration.
Implement a robust review process: Review Terraform configurations carefully before applying them to production.
Educate your team: Ensure that team members understand the importance of state consistency and the potential risks of stale plans.

By following these best practices, you can build a robust and reliable infrastructure automation pipeline using Terraform and GitHub Actions. Remember, infrastructure as code is a journey, not a destination. Continuous learning and improvement are key to success.

So, there you have it! We've covered the stale plan problem in depth, explored various solutions, and outlined best practices for using Terraform in GitHub Actions. I hope this has been helpful. Now go forth and build awesome infrastructure! Keep those plans fresh and those deployments smooth. Cheers, guys!