Move OpenSearch Workflow To Larger GitHub Runners

by Alex Johnson 50 views

Addressing Functional Test Failures in opensearch-k8s-operator

The opensearch-k8s-operator project has encountered persistent issues with its functional tests on GitHub Actions. The core problem lies in the limitations of the default GitHub-hosted runners, which lack the necessary resources to accommodate the demands of running a complete OpenSearch cluster alongside the upgrade scenarios within the continuous integration (CI) environment. These resource constraints manifest primarily as disk space exhaustion during provisioning and upgrade processes, leading to job terminations and hindering the project's development velocity. To resolve this, a strategic shift towards larger infrastructure is essential to ensure reliable test execution and streamline the merging of pull requests.

To ensure the stability and reliability of the opensearch-k8s-operator project's CI pipeline, a crucial step involves migrating the functional tests workflow to more robust infrastructure. The default GitHub-hosted runners, while convenient for many tasks, fall short when it comes to the resource-intensive nature of these tests. Specifically, the functional tests require spinning up an entire OpenSearch cluster and simulating upgrade scenarios, processes that demand significant disk space and processing power. The current setup frequently leads to disk exhaustion and job termination, causing delays and bottlenecks in the development workflow. This necessitates a move to larger runners capable of handling the load, thereby ensuring consistent and dependable test results. By addressing this infrastructure limitation, the project can unlock its full potential, enabling developers to merge code with confidence and accelerate the delivery of high-quality software.

The limitations of the default GitHub-hosted runners for the opensearch-k8s-operator functional tests stem from their inability to handle the resource-intensive nature of the OpenSearch cluster deployment and upgrade simulations. These tests, designed to ensure the operator's proper functioning, involve provisioning a full-fledged OpenSearch cluster, which consumes a considerable amount of disk space and memory. Furthermore, the upgrade scenarios add another layer of complexity, requiring additional resources to execute smoothly. As a result, the default runners, with their limited capacity, become overwhelmed, leading to frequent failures and hindering the project's CI pipeline. The constant interruptions and delays caused by these failures impede the development process and make it difficult to merge new features and bug fixes. Therefore, migrating the functional tests to larger infrastructure is not merely a performance enhancement but a critical requirement for maintaining the project's momentum and ensuring its long-term success. The transition to larger runners will provide the necessary resources to execute the tests reliably, enabling the development team to focus on building and improving the operator without being bogged down by infrastructure limitations.

Proposed Solution: Self-Hosted Runners

The proposed solution involves leveraging self-hosted runners, offering a tailored environment capable of meeting the specific demands of the functional tests. This approach entails provisioning a Virtual Machine (VM) or a small pool of VMs with specifications exceeding those of the standard GitHub runners. These VMs will then be registered as runners with specific labels, such as self-hosted, linux, and large, enabling targeted job allocation. By limiting the runner scope to the repository and dedicating it exclusively to functional tests, we ensure that resource-intensive tasks are handled efficiently without impacting other workflows. This strategic allocation allows lightweight jobs like linting, unit tests, and documentation builds to continue utilizing the GitHub-hosted runners, optimizing resource utilization and minimizing unnecessary costs.

Implementing self-hosted runners for the opensearch-k8s-operator functional tests presents a viable solution to the resource constraints encountered with the default GitHub-hosted runners. This approach involves provisioning a Virtual Machine (VM) or a small pool of VMs that boast specifications exceeding those of the standard GitHub runners. These VMs are then registered as runners with distinct labels, including self-hosted, linux, and large, which act as directives for targeted job allocation. By carefully configuring the runner scope and dedicating these resources exclusively to functional tests, we ensure that resource-intensive tasks are handled with optimal efficiency. This targeted allocation strategy prevents the functional tests from competing with other workflows for resources, ensuring that critical CI tasks are executed reliably and without interruption. Moreover, it allows lightweight jobs, such as linting, unit tests, and documentation builds, to continue leveraging the GitHub-hosted runners, thereby optimizing resource utilization and minimizing unnecessary expenditure. By adopting this hybrid approach, the project can strike a balance between performance, cost-effectiveness, and maintainability, paving the way for a more robust and streamlined development workflow.

By opting for self-hosted runners, the opensearch-k8s-operator project gains greater control over the testing environment, enabling it to tailor the infrastructure to the specific needs of the functional tests. This level of customization is not achievable with the default GitHub-hosted runners, which offer a one-size-fits-all approach. With self-hosted runners, the project can specify the exact hardware configuration, including the amount of CPU, memory, and disk space, ensuring that the testing environment has ample resources to handle the most demanding scenarios. This flexibility also extends to the software environment, allowing the project to install specific dependencies and configure the system to match the production environment as closely as possible. Furthermore, self-hosted runners offer the potential for cost savings in the long run, as the project only pays for the resources it uses, rather than relying on a shared pool of resources with variable performance. This control, customization, and potential cost-effectiveness make self-hosted runners a compelling solution for the opensearch-k8s-operator project's functional testing needs, paving the way for a more reliable and efficient CI pipeline.

Implementation Steps

  1. Provision Infrastructure: Set up a VM (or a small pool of VMs) with specifications exceeding the standard GitHub runner size. This should include sufficient CPU, memory, and, most importantly, disk space to accommodate the OpenSearch cluster and upgrade scenarios. Consider factors like scalability and redundancy when planning the infrastructure.
  2. Register Runners: Register the provisioned VMs as runners within the GitHub repository. Assign labels like self-hosted, linux, and large to these runners. This labeling system allows for precise targeting of jobs to the appropriate runners based on resource requirements.
  3. Configure Runner Scope: Limit the scope of the runners to the specific repository (opensearch-k8s-operator) to ensure dedicated resource allocation. This prevents other projects from inadvertently consuming the resources allocated for functional testing.
  4. Workflow Modification: Modify the functional tests workflow to target the newly registered self-hosted runners using the assigned labels. This ensures that the functional tests are executed on the larger infrastructure, resolving the resource exhaustion issues.
  5. Monitoring and Maintenance: Implement monitoring and maintenance procedures to ensure the health and availability of the self-hosted runners. This includes tracking resource utilization, applying security updates, and addressing any performance bottlenecks.

Benefits of the Solution

  • Reliable Testing: By providing sufficient resources, the self-hosted runners eliminate the disk space limitations that were causing frequent test failures. This leads to more consistent and reliable test results, increasing confidence in the codebase.
  • Faster CI Pipeline: With dedicated resources, the functional tests can execute more quickly, reducing the overall CI pipeline duration. This allows for faster feedback loops and quicker iteration cycles.
  • Efficient Resource Utilization: By offloading resource-intensive tasks to self-hosted runners, the GitHub-hosted runners are freed up to handle lighter jobs, optimizing overall resource utilization.
  • Unblocking Development: The resolution of the test failures unblocks the merging of open pull requests, allowing developers to contribute more effectively and accelerate the project's progress.

Timeline and Next Steps

While there is no specific deadline, the implementation of this solution is a high priority due to its impact on the project's development workflow. The sooner the self-hosted runners are provisioned and configured, the sooner the functional tests can be reliably executed, and the project's development can proceed smoothly. The next steps involve:

  1. Infrastructure Provisioning: Begin the process of provisioning the necessary VMs or VM pool.
  2. Runner Registration: Register the provisioned VMs as runners within the GitHub repository.
  3. Workflow Modification: Update the functional tests workflow to target the self-hosted runners.
  4. Testing and Validation: Thoroughly test the new setup to ensure its stability and performance.

This initiative is being tracked on the OpenSearch Project Projects board, providing transparency and allowing for collaborative monitoring of progress.

Conclusion

In conclusion, migrating the functional tests workflow for the opensearch-k8s-operator project to larger, self-hosted runners is a critical step towards ensuring reliable testing, optimizing resource utilization, and unblocking development. By addressing the resource limitations of the default GitHub-hosted runners, the project can achieve a more robust and efficient CI pipeline, leading to faster development cycles and higher-quality software. This strategic move will empower the development team to confidently merge code, accelerate innovation, and deliver a superior user experience. The implementation of self-hosted runners not only resolves the immediate issue of test failures but also lays the foundation for a more scalable and sustainable testing infrastructure, capable of accommodating the project's future growth and evolving needs.

For further information on GitHub Actions runners, visit the official GitHub Actions documentation.