Real end-to-end testing requires real-world transactions. Sometimes that’s not too difficult to do if you are implementing payments in your own country. But step outside your border, and it gets a bit harder.

People new to payments turn to logical solutions when faced with this problem. Can you call the acquirer? Won’t they have test accounts for us to use? (Probably not.) Can’t we just drip a few transactions through with a few live customers to see how it goes? (Sure, but you may wait a long time to run through all the scenarios you would like to see – and there usually isn’t patience for that in a launch environment.)

It’s tough, but don’t be discouraged. Mercifully, a subspeciality of payment providers has set out to solve this problem. And that is the focus of this episode.

Steve Semelsberger, CEO of Testlio, joins Yvette Bohanan and Drew Edmond to discuss the evolution of quality management alongside payments testing.

Yvette Bohanan: Hello, I’m Yvette Bohanan, a partner at Glenbrook and your host for this episode of Payments on Fire. Oh, testing. The bane of the payment engineer’s existence. Sure, there are developer toolkits and sandbox environments pretty much everywhere these days. And there are still people who believe that if you run payments properly in the sandbox, you’re good to go.

But those of us who have been around for a while know that’s just not true. Real end to end testing requires real world transactions. Sometimes that’s not too difficult to do if you’re implementing payments in your own country. But step outside your border and it gets a bit harder. Can you find a friend, or a friend of a friend, willing to make some payments for you?

Good luck. And you’ll need more luck if you want to test transactions like split tender, refunds, or chargebacks. People new to payments turn to logical solutions when faced with this problem. Can you call the acquirer? Won’t they have test accounts for you to use? Probably not. Can’t we just drip a few transactions through with a few live customers and see how it goes?

Well, sure, but you may have to wait a long time to run through all the scenarios that you’d really like to see and there usually isn’t patience for that in a launch environment. It’s tough, but don’t be discouraged. Mercifully, a subspecialty of payment providers has set out to solve this problem and that is the focus of this episode.

Joining me as co-host for this conversation is Drew Edmond, my colleague at Glenbrook, who gets this testing question a lot. Drew, thanks for joining me on this episode.

Drew Edmond: Thank you for having me once again. I’m excited to talk about this topic. It definitely impacts every client that we work with, whether or not we’re working with them directly on it or not, it’s a topic that impacts our organization because everyone has to or should be testing their payments integration.

And I remember in the early years when I was at Square, we used to actually, just in the smaller office, we would hold testing parties in the conference rooms with just employee volunteers, folks that were around the office and had their phones with them, which we all did, of course, at the time, and we would just sit around and basically just go through various testing flows because we wanted to identify bugs and report those back to the engineering team and test the end to end payments flow and the customer experience. And it was really useful. But everyone had to use their own debit card or their credit card and sometimes we would go out and buy some prepaid cards at the CVS down the street or something and bring them back in. But then everybody that did that had to go off and individually work with an engineer to go get those payments refunded and all that.

So it’s not this seamless experience. It was very helpful and it was necessary, especially as a growing financial services business that we wanted to have great experiences for people and things like that. So it was not optimal, it was not comprehensive, it certainly wasn’t global.

Yvette Bohanan: Or scalable.

Drew Edmond: We would have to just keep hiring more employees if you wanted to expand it, which maybe isn’t the best reason to hire just on that alone. So yeah, when I think about merchants and the payments companies that we work with, I think that the testing completeness and their practices related to it, I’m sure varies quite a bit, and it certainly does.

And I think that there just maybe isn’t a lot of awareness about the solutions that are out in the market today. So I think it’s just a great opportunity to bring our guest in and to get the word out about what folks can use in this space.

Yvette Bohanan: Fantastic. So let’s do that. Let’s introduce our guest. So we are absolutely delighted to have Steve Semelsberger, CEO of Testlio, joining us on this episode. Steve, welcome to Payments on Fire.

Steve Semelsberger: It’s great to be here. Yvette, Drew, thanks so much for having me.

Yvette Bohanan: We always like to begin a podcast by asking folks how they arrived at their current role in payments, and I really do want to hear about your journey, but before we go there, I have to ask a slightly personal question. As a child, did you enjoy breaking things?

Steve Semelsberger: So I do work for a quality management company. You would think that I was a breaker. I’m not the founder of Testlio, though. Our founder is Kristel Kruustuk, who you should get to know. She is amazing, wonderful, a visionary in the space, and she was a freelance software tester who saw the opportunity to create a global company to tackle some of the world’s biggest testing challenges.

And yeah, she was curious and messing with stuff all the time. I think as a kid, Yvette, I was, I was an integrator. I was a pluralist. I was a generalist. I was the kid who was skateboarding, playing football for the high school, in a band. If you’re seeing this on video, there’s a drum set.

So I tend to love synthesis and integration and finding ways to bring complex concepts together. And what I often think of is a non-dualistic approach and we’re going to talk about the non-dualism possible in quality engineering today. And so I’m probably more of a builder than a breaker, Yvette.

Yvette Bohanan: That’s a great way to describe it, so as a builder and not a breaker. You were telling me a little bit as we were preparing for this conversation that you invested in Testlio early on. And here we have this visionary female founder of this company with tons of experience, obviously, if she was freelancing in the space and you come along and you decide to invest. What made you decide to invest? What made you put your energy into building Testlio?

Steve Semelsberger: So I met Kristel and her co-founder Marco, now husband, married, two beautiful children. They’re from Tallinn, Estonia. They won a global hackathon competition, Yvette. So there’s something called AngelHack. They were in London, they won London, they were brought to San Francisco, they won it. And there’s an accelerator in Austin called Techstars.

And the MD of Techstars, Jason Seitz, found Kristel and Marco just with basically this idea and this prototype that they’d hacked together and said, come to Austin, be part of my accelerator. 2013, first Techstars Austin accelerator. So I met Kristel and Marco through the Techstars process where I served as a mentor and I got to watch them over about a year, really unlock the vision that they had for the company.

And they are this amazing co-founding combination of very grounded, very powerful, clearly passionate, visionary in Kristel, and then Marco is a technical builder. So he is an engineer who can also do UX and visual design. And so the two of them were this perfect combination and I thought they were entirely authentic and I thought that they were tackling a really big opportunity for efficiency and improvement. So globally, companies spend about 80 billion dollars a year on what most would call quality engineering today. We could talk about software testing, quality assurance, quality engineering, and how we use those terms.

But if you define this landscape, it’s a lot of money is spent. And in many cases, that spend is not very efficient nor effective. And Kristel and Marco had a idea on how you do software testing better and I loved it. I invested, I advised for four years, and it was one of those stories where you love the company so much that you find a way in.

And you mentioned Kristel as a female founder. Our business is 46 percent women with our full time people. We’re a very global company. We have freelancers in 165 countries, and we have full time people in 43 different countries. And so we really create cultures of inclusion. One of our values in the company is foster inclusion.

And so I worked really closely with Kristel to make sure that she was truly ready to step into more of an evangelist role with the company as a male CEO coming in. I didn’t want this to be a moment where the investor was like pushing out the founding CEO. It hasn’t been that. And I think that’s been key to the six year journey that we’ve had together.

Since then, Kristel was clear about what she wanted. She created space and she does what she loves. And I’m good at like board decks and budgets and figuring out commission plans with the sales team. And so we really complement each other nicely.

Yvette Bohanan: I’m going to guess that you know a couple things about testing though now too that we can talk about so let’s dig in. I appreciate you sharing all of that actually because you said some words that really resonate. First of all, that you were a mentor. I think some of the most enjoyable things I’ve done in my career were around mentoring.

I was a mentor at 500 Startups for several years and the enthusiasm and the authenticity and the energy of the people that have this vision and whatever it is and what they’re trying to achieve is just so much fun to be around. And then getting to take that and grow it and scale it into this organization is just fabulous.

Steve Semelsberger: It really is. You’ve probably experienced this Yvette, but as a mentor, I often felt like I was getting much more than I was giving and Techstars as an organization, the motto is give first. So everybody is in this altruistic mode and it’s really a beautiful and fun and to your point, energizing thing when you’re a part of it.

Yvette Bohanan: Yeah, I absolutely agree. 110%. We’re going to talk testing though. So let’s start at kind of a higher level, put payments aside for a moment. Has testing, whether you call it QA or QE, quality assurance or whatever, has it become more or less complicated over say the last 10 years?

Steve Semelsberger: So you remember a minute ago, I said, I’m a pluralist.

Yvette Bohanan: Yeah.

Steve Semelsberger: I like to look at things in non-dualistic senses. So my answer is, it’s both, Yvette. It’s actually both. Stepping back, the way that I would offer software testing as a definition is a set of things that companies do to generally try and find bugs that can be remediated. Quality assurance is a step further in that it covers software testing and it also attempts to make sure that issues are addressed before you push something generally from, say, staging into production. I would then say quality engineering is an even bigger concept. I wrote an inverted pyramid article where I put software testing at the bottom, the small point of an inverted pyramid, and then I argue quality assurance is bigger.

Quality engineering today, I’d say, is an even bigger idea, and it has to do a lot with the evolution of the intersection of DevOps and quality assurance. And so how do companies go really quickly and how do they release with high velocity and lots of confidence in those releases? And how do you make sure that whatever it is that you’re testing, that you’re testing it intelligently and in an economical fashion?

And so there’s all sorts of amazing new AI powered tools and methodologies and processes. And because of the options available, it’s really complex. It’s hard for companies to figure out what to do and how to do it. So maybe I dodged your question, Yvette, but we could spend the whole podcast talking about this key point, because it’s a hard time for business leaders as well as engineering leaders.

And sometimes I find it’s a hard time for engineering leaders to explain it to the CFO and the CEO as to like what they’re doing and why they need to do it. Because it’s easy sometimes, I think, to go, Oh, QA is a cost center. Can’t we just reduce our spend on it? Won’t we go faster if we don’t do some of these things? And the answer is yes. And there’s some repercussions to that as well.

Yvette Bohanan: How do you suggest the engineering, product, QA people engage with leadership to explain that inverted pyramid and the benefit of it?

Steve Semelsberger: Yes, I think it starts with having a clear viewpoint on your quality engineering strategy. Our chief client officer Summer Weisberg wrote a LinkedIn post on this recently. I referenced it, expanded it a bit. I have a current series on LinkedIn called QE for CEOs and it’s really meant to help VPs of engineering, who are our primary clients, figure out exactly this.

Strategy. Yeah, that’s another one of these juicy topics. What’s a strategy? What’s a tactic? How do you tie your strategy to the objectives of the organization? How do you manage through uncertainty? Yet if you have a clear strategic position on quality engineering and the organization buys into it, so it’s got to be like right and simple, it’s hard to do, but when you get it, sort of like a vision statement for a company, if you have your strategic direction aligned, you kind of know it, then you make a case for your level of spend. And what we’ve seen in the market is that companies who are committed to quality engineering are spending between 10 and 20 percent of their software development all in fully loaded spend. So if you look at what it costs for you to have software developers, you then want to allocate 10 to 20 percent. And it varies. For some companies, it’s literally zero percent and for others, it’s 40 percent. There are a lot of different factors that go into this, we can get into them if you’d like. But you can make a case as to how you will spend and then the kinds of results that you can see through that level of spend. And systems and tools and analytical platforms, things are getting so good now that you can constantly test that spend and determine if it makes sense.

Yvette Bohanan: Interesting.

Drew Edmond: So Steve, the, the name of this podcast is Payments on Fire. It’s not Software on Fire.

Steve Semelsberger: I feel like software on fire sometimes.

Drew Edmond: Yeah. Sometimes software is on fire. Just to take it down the payments path a little bit, because there’s so many nuances in the world of payments.

I’m curious from your perspective, when you added the payments component and what you’ve learned from the added dimensions really that set payments testing apart from general software, general application testing

Steve Semelsberger: It would be cool to say like, originally Kristel had a vision to become the biggest and best payment testing company on the planet, but that wouldn’t be true. So we actually did our first real world transaction test over 10 years ago and since then, we’ve done over 200 payment engagements.

Our customers process a lot of money in different directions. We think it’s about 5 trillion dollars a year. We think that based on some feedback we’ve gotten, we may have just done more payment testing than anybody. And we’ve gotten to this, though, through lots of other work. We work with streaming media companies. We work with sports leagues like the National Basketball Association. So our business is really a quality management company. It’s a technology enabled services company. Yet the fastest growing piece of this is around digital transactions. I think the reasons are pretty obvious probably to you and your listeners in that 2 out of 10 digital transactions fail.

It’s really sometimes very hard to figure out why things fail. Companies are constantly tuning and doing things. So I love, Drew, when you describe what you did at Square. I mean, that’s what a lot of companies have been doing. And in many cases, that actually works. Yet, Yvette, when you were listening to Drew’s story, you said something about scale, right? How hard it is to do internal hackathon kind of, on a weekend, buy people donuts and ask them to, that’s really hard to do with consistency and rigor. And things are moving so fast that most companies today who have been doing things like you were doing, Drew, at Square, are exploring more.

They’re really thinking about burstable on-demand approaches that are economical and that match machines and humans in ways that allow them to move fast, not spend a lot of money, and also have more predictable results that helped them get to the root cause of potential issues.

Drew Edmond: Totally. And I think just with the growth too, global businesses, global merchants, and the breadth of things like payment methods, right? It’s not like we’re just doing Visa and Mastercard payments. We’ve got all these card brands, we’ve got all these different payment methods, the number of devices that exists across the world, the number of actions that you want to take and test as a merchant. There’s no way to do that with 10 people sitting in a room.

Steve Semelsberger: You’re so right. We think we’ve done over or tested over 800 payment methods in our history of doing this and we’re adding more all of the time. And for many of our clients, it’s the intersection of method, device, sometimes language, and certainly location.

And so, for example, I mentioned we work with sport leagues, the National Basketball Association has a major commitment to Africa. Africa is a really important market to the NBA and they had a tricky issue happening in Nigeria. And it was the kind of thing that people in a lab or using pure automation really couldn’t figure out. And so the approach there is to have people in market on in the wild devices with a variety of local banks connected to various gateways and instruments. And when you’re in the wild doing these sorts of things, you learn a whole lot through the process. And so then how do you take the learning and the insight and turn that into something that a software development team can quickly remediate, sometimes themselves or sometimes in combination with providers, vendors, and partners.

Yvette Bohanan: Do you get involved in reporting issues back to the networks or to the providers on behalf of your client?

Steve Semelsberger: So sometimes the provider is the client. We publicly disclose that we work with PayPal. We work with other network providers that we’re not at liberty to publicly disclose.

Yvette Bohanan: Right.

Steve Semelsberger: So the network providers care a whole lot themselves. To your point, Yvette, there’s this collection from commerce subscription, SaaS, and other endpoints that are trying to process these payments in a whole variety of ways.

So the good news is, our team was at Money 20/20, and like people care about this. They really care because not only is it an economic problem, but it’s an end user experience problem as well. Companies want their users not to get frustrated, certainly when they’re trying to either put money in or actually take money out.

So we do quite a lot of work on two sided markets. Companies like Uber and Etsy are our clients. And in that case, actually making sure that the driver gets paid, that the merchant gets paid, that money processes out is really important. And we work with Western Union.

It’s that connection between humans all over the planet. So the cross-border aspect, the in and out of digital, and we could talk about crypto as well too, because that also adds another level of complexity at times too. So the whole thing is just, if your audience is feeling like it’s hard, it is hard. It’s all really hard, but the good news is there are more potential approaches that you can try than ever before.

Drew Edmond: All right, Steve. So as of the time of this recording, Bitcoin is essentially reaching all time highs on a permanent basis, it seems. And stablecoins are in the news a lot and becoming a major form of global money movement, really moving double digit trillions of dollars these days or over the last year or so.

When you think about digital currency systems like crypto and stablecoins, what changes are you testing in these payments and how do you approach testing these payments that operate a little bit differently than maybe what we’re used to?

Steve Semelsberger: We found when we’re working with blockchain centric, financially oriented companies, so two companies that were on record who are clients our BitPay and Uniswap, and in that case, we’ve had to make sure that the teams of people who we use, so we use a combination of our full time people, as I mentioned before, and freelancers, that they deeply understand blockchain technologies and that the testing methodologies and approaches and tools that we’re using are really tuned into the nature of how the software is built. So Yvette, I know you’ve had a career in engineering and Drew, you’re deep. You know what white box and black box testing is?

Yvette Bohanan: You should explain it for our listeners though.

Steve Semelsberger: I’m not an engineer, but I’ve worked in software for 30 years now. But my basic thought is that when you’re doing black box testing, you’re not really aware of how the technology works.

You’re looking at the functions and visual experience. So you can do that through machines and humans. A lot of times the combination gives you great results. There’s gray box testing where you have partial awareness of how things come together. So some end to end testing, some people think of at times using API endpoints as gray box.

And with white box testing, you’re deep into the mechanisms of how the code comes together to create a software workflow or experience. And so I think depending upon the level of access that your testing function has, depending upon the complexity and the goals of the engineering organization, taking a combination of white, gray, black, can also really help out. Yvette, did that answer your question?

Yvette Bohanan: Sort of. I think. Maybe.

Steve Semelsberger: I took it kind of in a different direction.

Yvette Bohanan: I get where you’re going with your response, for sure. And I guess that’s kind of where I was going with my question, right. It is more of a black box environment to people using these systems. You can see what’s in your wallet, you have the experience, but you really, unless you’re a node on a chain, you don’t really know the code in that. And I guess it’s kind of true as I’m sitting here saying this out loud about, we don’t really know the code that our bank is running that debits and credits our accounts. So I guess maybe it’s not as different because the techniques are similar, it’s just having the experience of knowing how to use the system, and that’s where your testers come in.

Steve Semelsberger: I think you’re spot on, Yvette. And some of it goes to the scope and scale of the organization. There’s a large US bank, they’re not a client of ours, but my understanding is that they have thousands of mobile developers for their consumer checking and savings account application experience.

And so you can imagine when you have thousands of iOS and Android developers, let alone the team that’s building the back end systems and integrating things, that the resources and precision and opportunity to go really deep all the way down into your unit tests, what individual developers are creating through integrations, features, and functions, are massive.

And it’s one of the reasons why those apps are so great, that the resources are really high. And for companies who have 10 engineers and maybe series B funding, the approach that you take is really different. Generally the app is narrower, often the stakes are less high and sometimes there’s also an element of tolerance as well. The need to have high confidence in a release versus a willingness to try some things and move quickly, it is really important when you’re thinking about your quality engineering strategy.

Drew Edmond: When I think about going back to my example of sitting in a conference room, if I saw something strange happen, I could just walk over to an engineer and say, that looks weird. What’s going on there. And they can look it up and we could try to figure it out together. When you’ve got people spread out all over the place in different languages, and the process to both construct the testing as well as the framework for how that information is collected and brought back so that it’s useful to the business has got to be critically important, I would imagine.

So I’m curious, could you just tell us how that works? What makes it useful to the business in that way?

Steve Semelsberger: Yes. One of the things that Kristel saw early on that I thought was really interesting was what many will call burstable software testing. So just like EC2 and cloud services give you burstable compute, the opportunity in the market today is to have non-standard fixed capacity. And so a big problem with quality is this sense of like, I don’t have enough people or machines when I really, really need them.

And so an approach that you can take is to literally tune lots of resources at the moment of need, and then allow those resources to go dormant. And sometimes that’s fairly predictable and rhythmic. So for companies to release iOS apps into a store every two weeks, you can say every Thursday, we’ll have a bill that’s a candidate production release and we’re going to have 50 people, two hours each, attack it. And you can have scripted tests, automated smoke tests, exploratory tests, it can all happen really fast. With global labor, cool thing too is the bill can be done on a Thursday night, the testing can finish up midnight Thursday local time, results can be in the next morning for the engineering team.

There’s no business hour delay, even though there’s some clock time delay. And then those resources go dormant until they’re needed again. And you can also do that in local markets. So we work with companies like PayPal in 50 different countries. We work with a large social media company in 75 different countries.

And their localized releases happen only a few times per year, but they want to make sure that when a localized, linguistically translated, as well as often functionally specific version of that product is available, Arabic in Turkey, making sure that it’s tuned for the Turkish market, that there are people on the ground to do the work.

And so sometimes that burstability is every two weeks. Sometimes it’s like rhythmic throughout the course of the day. Sometimes it might be every four months. And so making sure that you have these resources available when you need them is where software and methodology meet.

So you can have a great process, but in order to do the kinds of stuff we’re talking about, it’s a logistics problem. So Summer, who I mentioned before, who runs our client services team, she spent 13 years at FedEx. And you can think of like shipping packages around the world and making sure that they’re tracked and they get there.

We do similar things around inviting freelancers into work because by definition, they’re freelancers. You can’t assign them work, you invite them work just like you make opportunities available for an Uber driver. So these invitations have to find the right person on the right device in the right location who’s fully qualified and vetted.

We only accept about 3 percent of freelancers who apply to do work with Testlio. So again, go back to Kristel, she had a very high bar. Bring the best people in the world together, give them awesome opportunities to work with some of the biggest and best companies in the world, pay them well, and then equip them often to do work in times that are really great for them.

So a mom at home in the evening after kids have gone to bed, here’s a two hour quick invitation for you. You could do that out of your home office in many cases, and really help clients in a way that’s good for them and again good for the whole system.

Yvette Bohanan: I think this is all really fascinating, actually, because you’re managing quite a workforce. But I’m going to double down on what Drew was kind of getting at, I think, which is, and maybe this ties into how you think about sort of software testing assurance and quality engineering, right, in this inverted pyramid. If I was sitting there as the CTO of imaginary company, whatever, I would want to know if my strategy for quality engineering was working or not. So I would be benchmarking failures per whatever million lines of code or are we reporting more or less errors on the initial release in Turkey or all that kind of stuff, right?

With a freelance invite environment, how do you ensure that your data that’s being collected is consistent and reliable and you can generate trends and metrics that you can really, pardon the pun, count on, right? Versus, well, this group of testers did it this way in Turkey last time, and we had a different sample size, so it’s really hard to compare. How do you get there to that sort of nirvana that you were pointing to in engineering, quality engineering?

Steve Semelsberger: So, I think you said something really important, Yvette, around the metrics that you use and the way that you measure things and how that’s tracked and instrumented into workflows and processes and available in third party systems as well as natively in testing management tools like the Testlio platform that we provide to our clients.

So we give our clients full transparency, access to everything that’s happening in our systems, unlimited licenses, we don’t charge for our software. So it’s all really, really visible. They can see the people that are doing the work, why they were invited, what their ratings are, how they’ve performed in the past.

And then using a combination of machines and humans, we really push hard to make sure that something that we think is an issue has the right level of severity viewpoint on our side, and then often gets pushed into the Jira or other issue tracking system at our clients so that they can then prioritize based on severity. What we found too is that Gen AI really helps. So, we use the Microsoft Azure OpenAI endpoint. And because we segment all of our work for our clients into different workspaces, so, you imagine that the console that people use to interact and view and see reports, we segment that out. One of the things we love about Microsoft is they ensure that any client content that we put in is extractable and is protected, you know, versus throwing it into consumer grade ChatGPT.

And so then, okay, things are going in and so how are we using AI? There’s three things that are working really well right now.

The first is around test case creation and test case refactoring, so the instructions that are built to do the work. When you use generative AI, our time and motion studies show that it’s between 15 and 40 percent faster than if a human creates those things alone. Then, when somebody believes there’s an issue, you run the issue description through analysis to see if there are potential dupes, as well as you use AI to help people create an issue that’s understandable, that is aligned with the company’s standards, using various template formats, et cetera, and it then is viewed as clear, concise and trustable. And what we’re seeing is that issues that have AI assistance tend to be acted on more. It’s correlation, but it seems like, oh, if this is a really wonderfully articulated issue, I’m much more likely to investigate it, trust it and take action against it.

Yvette Bohanan: Interesting.

Steve Semelsberger: Yeah, with our clients, there’s two things that we work really hard, we work hard on a lot of things, but two metrics to your point from before, Yvette, that I encourage others to think about too. One is escaped issues. So being ruthlessly honest about anything that seems to be making its way into production for whatever reason, why did it escape?

And then what are your forensics and your methodologies to attempt to get escapes to zero. For many companies, there’s no tolerance, zero escaped issues, especially around payments. And then also for anything that’s caught, hopefully pre-production, what’s the addressed issue rate.

Because if you’re finding issues, but they don’t really matter, if the engineering team doesn’t want to do anything with them, then it’s kind of like, well, what’s the point here. And then you can look, to your point from before, Yvette, around issues per run, issues per dollar spent on testing, issue reduction per build over time, and really looking at time based patterns too as companies are going through different rhythms of build, release, improve, eliminate tech debt, gets interesting as well, too. There’s literally dozens of metrics to look at and all of them can help organizations tune and get a sense as to whether or not it’s really working or what they might wish to experiment and change.

Yvette Bohanan: Makes sense. Makes sense. That’s so advanced compared to where, you know, we were talking about scalability before in terms of, what did you say, giving people donuts and asking them to come in on Saturday for a bug a thon or whatever, a test a thon. But you’re taking it, you’re basically saying with the right tools, you get to sort of the metrics that actually drive business decisions. Is that part of what the quality engineering level of achievement is really all about?

Steve Semelsberger: Well put. Yes.

Yvette Bohanan: Interesting. Interesting. I think a lot of people have tried to get there before and just couldn’t get over the hill, right? The complexity was going up faster than the ability to scale the testing and track it.

Steve Semelsberger: That’s right. It’s hard. It’s hard when you’re a big company and you have lots of resources. It’s hard when you’re a small company. It’s really hard. What I really adore about our clients is that there’s this desire to constantly improve. And it’s one of the things I love about my team too, is if you feel like, Hey, we’ve got a regression test, that’s pretty stable and it seems to be doing what it’s supposed to do. That’s like a moment in time. It’s like ecosystems, they seem balanced, but they’re not. They’re constantly dynamic and moving. And I think software development and quality engineering associated with it is very similar. It’s constantly in a state of flux based on people, tools, process, business conditions, psychology, globality, devices.

We could talk a whole lot about even hardware specific testing. For some of our companies, streaming media companies, for example, there are 12 OTT devices that they support. So you have to work on the Chromecast and the Xbox, et cetera. And so making sure then that that experience is tuned to the hardware that your users care about and in places where Android devices are prolific, or less expensive Eastern manufactured devices, such in certain African countries are specific, you then get into all sorts of challenges that you may not have on the latest iPhone.

And so for some companies, their hardware coverage strategy, coupled with location, language, payment method, and then testing methodology all need to come together and constantly be evaluated.

Drew Edmond: In terms of the output that you get back from the testers, from spreading across qualitative experiences, quantitative data that might be measured during the course of it. Is it that the data is like actually being recorded on their device and something’s being provided back? How are they capturing that information and structuring it?

Steve Semelsberger: So data can be recorded. Things can be captured when there’s an issue. So in some cases, do you instrument the experience, you’re recording everything? Or once an issue is discovered, do you then make sure that that’s fully recorded? Some clients prefer screenshots, some want full video recording.

Typically, best practice is seek at least three reproductions of the issue. And again, ensure that it’s not a duplicate issue. And reproductions can be done. And sometimes it’s a machine that catches something that a human reproduces it, sometimes it’s human, other humans. Sometimes you try to repro against other devices. Sometimes you want the specific device to hold too.

So showing then that this is not just an edge case that a single person had or a single, let’s say, automated Selenium script may have discovered, but this seems to be something that we can repeat. And then again, determining what it might be from a severity standpoint, from our perspective and how customers might prioritize that, then moves into the notion of how quickly do we fix it? And sometimes it’s minutes turnaround for the whole thing. And sometimes it’s days and weeks.

Yvette Bohanan: All right, so we’re coming up on time here, Steve. This has been very interesting and illuminating on a lot of levels and hopeful too. So this is good because people need help. Let’s say I’m a listener right now and I’m doing payments software testing. Okay, I’m in that small part of your pyramid and I want to move into QA, just assurance level.

What are two to three things I should think about doing just to move up to that next section of the pyramid?

Steve Semelsberger: Yvette, should we focus on the payments piece?

Yvette Bohanan: Yeah, let’s bring it home to our payments audience for now.

Steve Semelsberger: What I would encourage companies to do is as a second step in that equation. So let’s pretend the fictional client of Glenbrook’s is where Drew was with Square. So we kind of test payments, but it’s sporadic. Pick a reasonable timeframe, whether that’s once a month, once a quarter, twice a year, and do a real transaction, human production payment sweep, and pick a reasonable set of devices and software interfaces and payment methods that you think matter and start with, a few times a year, people in the wild. That’s a really good first step, but we see a lot of companies to take that next. And then if you wanted to push it further, you could start to think about, okay, well, what payment methods, systems, processes do we test for pre-production?

You could then say, okay, well, we’re going to do a subset of this full sweep once a quarter with every regression, and companies run regressions sometimes multiple times a day, sometimes once a week. So you think about smoke testing real payment challenges.

So then I think the third piece is as you see where you might find opportunities to go faster, I’d say you can automate some of those pre-production payment tests and then by automating some of those you can push for exploratory testing in the wild, you can actually expand the number of payment gateways and methods and give humans room to like move around and try things. Because many of us experience the pop up interstitial intersection of third party, trying to determine, Hey, are you really you in this location on this device? We don’t see a cookie. You seem to be a different place. Two factor authentication, pop ups, interstitials, all of that. You want to make sure you understand the user experience and determine how much of a challenge that is.

So those would be three steps. A bit sporadic in the wild, start to then really do smoke testing on payments, pre-production on your regressions, and then finally pushed to have a much more comprehensive exploratory approach. There’s like a fourth and fifth step that we could spend more time on, but you’ve been generous with your time with me already today.

Drew Edmond: You brought up something at the end there that actually we could probably spend another hour on, which is fraud related testing because you mentioned some kind of signals there where it’s like, are our tools even set up correctly? Right? And maybe that’s something that you regularly want to be checking in on, where you’re saying, all right, let’s at least throw some out here, see what comes back, see what gets triggered, and make sure our rules are set up, or our fraud tools are configured accurately. I think that’s a really important point.

Steve Semelsberger: It’s a big, big issue. So identity fraud at time of initial identification and then also with things like national documents that expire. And so if something changes about the national documents, somebody updates their driver’s license or their passport, and it doesn’t seem to be a match, or there’s a name change, and sometimes that then prevents the transaction. So it’s valiant, great work to prevent fraud, but sometimes it’s understanding of identity and these other factors that then goes a bit too far and frustrates the end user and prevents the company from completing the transaction. But I guess, Drew, that’s, that’s our next conversation.

Drew Edmond: There you go.

Yvette Bohanan: Okay. And an important one. I think that’s one of the dimensions that’s been most overlooked in payments QA, honestly. We always say payments is a team sport, and what you’re really highlighting here too is, oftentimes, it’s a leader in the organization who’s saying to the team that’s responsible for risk management and fraud management operations, Great, our fraud’s really low. That’s wonderful. How do I know I’m not blocking too many good customers? And you’re highlighting something that’s really important. The communication between a comprehensive testing approach can really help that team have the insights they need to be able to say with confidence, We’re stopping the bad stuff, and we’re letting as much of the good stuff in as possible.

But you’re getting to that level, you know, when we talk about this in our risk management practice, that’s sort of the brass ring, nirvana moment, whatever, unicorns and rainbows moment of the perfect payments risk management environment and strategy is when you are optimizing everything, so all the bad stuff’s out, all the good stuff’s in, as much as possible. And that’s aligned with the QE strategy, right? So we’ll leave it there for our audience to ponder, I think.

Steve Semelsberger: Ponder, ponder away.

Yvette Bohanan: Ponder away. How do you bring those organizations together to really, really maximize what you’re doing here and joining forces within your organization to really move the needle in the right direction together?

Steve Semelsberger: Just like payments is a team sport, testing, quality assurance, quality engineering, they are very much team sports. Yeah, I think gone are the days, it may be if you’re a very, very small company of the lonely QA person in the corner. This is an era of technology and humans coming together with flexibility and burstability. And it’s hard, but when you get it right, it’s so cool to see it in action.

Thank you, Yvette and Drew. I really appreciate the opportunity to be here today. It was a joy speaking with both of you. So thank you again.

Yvette Bohanan: It’s our pleasure completely. Drew, thank you for co-hosting. We have a lot to think about now with our clients when it comes to testing and QA and QE. And to all of you listening, especially those of you who are tasked with testing, which apparently there’s a lot of you out there now, thanks for joining us and until next time, keep up the good work. Bye for now.