Every project leader knows that testing is where the rubber meets the road or when theory meets reality. It’s the stage that validates months, if not years, of discovery, planning, design, documentation, and development. Yet, testing is often and unduly underestimated, that is until something goes awry.
Why Testing Is One of the Most Difficult Phases
Testing isn’t just about checking if things work; it’s about ensuring they work under real-world conditions. It requires coordination across teams, operations, environments, customers, and timelines. This complexity grows when deadlines loom and stakeholders push for speed at all costs. This is where project leadership truly matters as it is easily noticed when balancing urgency with caution to provide measurable results.
The Temptation of Testing in Production
When timelines tighten, the idea of testing in production (prod) can seem appealing, and perhaps impactful. After all, production is the ultimate truth for all environments.
- Prod has real data
- Prod includes actual user accounts
- Prod runs with licensed tools
- Prod extends to all third-party integrations
- Prod reflects the full ecosystem your solution will live in
Testing in production can feel like the fastest way to validate everything. But here’s the reality: it’s perilous.
Why Testing in Production Is Risky
Production environments are not designed for trial-and-error situations that are truly unpredictable. A single misstep can lead to:
- Service outages that impact real users
- Compliance violations if regulated data is mishandled
- Reputational damage that no rollback can fix
- System corruption with a need to refresh from a backup
- Data loss and exposure of sensitive content (PII or PHI)
- PII = Personal Identifiable Information – Privacy related
- PHI = Personal Health Information – Privacy related
Unlike a controlled test (or other pre-production) environment, Prod is not built for and subsequently doesn’t forgive mistakes with any level of reasonability. Every action taken in Prod has immediate consequences and everyone needs to be aware of that constantly.
The Paradox: High Risk, High Impact
Sarcastically, testing in production can also be the most impactful with regards to measuring the results of inducing new code. Why? Because it validates the entire chain within the system including licenses, integrations, workflows, accounts, privileges, and user experience under authentic real-world conditions. It’s the closest you get to reality because the only variable is the newly injected code.
But impact and carefully written test scripts don’t justify recklessness, in my opinion. If testing within production is unavoidable, it demands the following in advance of the action.
- Rigorous planning
- Approval from the system’s operational owner
- Risk mitigation
- Clear rollback strategies and process outlined
- Clean and functional repository of the live code
- Consistent backup version of the live data
- A time when only the testing impacts the system, without live customers in the environment
- Acknowledgement from organization leaders that it is being done
Communicating Testing Risks and Results
Effective project leadership isn’t just about managing the technical aspects of testing as much as it is about clear communication both up the ladder and across each rung. Project Leaders and their development partners must ensure that all stakeholders understand the risks associated with different testing strategies. It is especially paramount that the ramifications of any potential testing in Prod are communicated. This messaging includes transparent reporting of test outcomes, potential impact, and contingency plans. By fostering open dialogue, leaders build trust and enable informed decision-making, reducing the likelihood of surprises and misunderstandings when issues arise.
Alternative: Building a Real-World Testing Environment
One of the most effective ways to avoid the pitfalls of production testing is to invest in a robust simulation environment for real-world testing experiences. This means replicating the production ecosystem as closely as possible by mirroring the following.
- Data structures
- User roles
- Integrations
- Content
- Servers and storage
- Delivery method
- Traffic patterns
While it’s rarely possible to achieve a perfect clone, the closer your test environment is to reality, the more reliable your results will be. Leaders should advocate for resources and time to build and maintain these environments, ensuring that teams can validate changes with confidence before anything reaches live users.
Checklist for Safe Testing Practices
Before approving any testing of new code, whether in production or a pre-preproduction environment, project leaders are best to ensure the following steps are addressed.
| Gain Approval to Review | Obtain authorization from leadership, operational owners, and relevant stakeholders approve that evaluation of testing can begin. | |
| Define Clear Objectives | What is being tested, and what outcomes are expected? | |
| Assess the Environment | Confirm whether the environment (production or other) is appropriate for the test’s risk level. | |
| Plan for Risk Mitigation | Identify potential risks and document mitigation strategies. | |
| Establish Rollback Procedures | Ensure clear, tested rollback plans are in place in case issues arise. | |
| Backup Critical Data | Create and verify backups of all affected systems and data. | |
| Schedule Testing Thoughtfully | Choose a time that minimizes impact on users and business operations. | |
| Ensure Real-Time Monitoring | Assign team members to actively monitor system behavior during testing. | |
| Document and Review Results | Record outcomes, issues, and lessons learned for continuous improvement. | |
| Write Communication Strategy | Note everything needed to inform all impacted parties about the test’s timing, scope, and potential risks. | |
| Obtain Approval to Proceed | Secure sign-off from customers, leadership, operational owners, and relevant stakeholders. |
By following this checklist, leaders can help ensure that testing is conducted safely, transparently, and with minimal risk to the organization.
Personal Success Story for Testing in Production
Epic Technical Dress Rehearsal (TDR)
One of the most memorable successes in my project leadership journey was during an Epic Technical Dress Rehearsal (TDR), where we were required to test in the actual production environment to ensure everything was ready for go-live. The stakes were high and every PC (all 20K) across the organization needed to be added correctly within Epic, and each workstation had to be set up with the right devices to support its specific aspect of the EHR system.
This Effort was Successful for Two Key Reasons
Exceptional Teamwork
The way our team handled the TDR was nothing short of fantastic. Every member worked to understand their role, we communicated clearly, and collaboration was seamless over Microsoft Teams. We anticipated challenges with the guidance of Epic, team members supported one another, and we maintained a positive, solutions-focused mindset throughout the TDR process. This collective effort ensured that all devices were configured correctly and every detail was double-checked.
Safe Production Testing
Although we were working in the production environment, the system was not yet “live” for actual patient care. This unique window before go-live allowed us to validate the entire setup including hardware, devices, accounts, peripherals, and integrations without any risk to real patient data or impacting of operations. It was the perfect balance: the realism of production, but with the safety net of no live impact.
By the end of the TDR, we had full confidence that every PC and peripheral device was ready for go-live. The experience reinforced the value of thorough preparation, strong teamwork, and the importance of leveraging production testing opportunities when they can be done safely.
Leadership Takeaway
Project Leadership here means saying “no” when necessary and “yes” only with safeguards in place. Testing is where projects succeed or fail quietly. As a leader, your role is to champion safe, structured testing practices. Resist shortcuts that compromise stability. Remember: the cost of failure in production far outweighs the convenience of skipping proper testing.