My Dear Friends of Figma in all right um I am French we French people have a terrible tendency we tend to point out things that don’t work so well so this is what we’re going to start with today right um there’s this problem that we’re facing in the design system industry at large um it’s been very well articulated.
By Amber finerman who is a senior product designer at Netflix in the design system team and she said that she would love to see more design system teams test and validate design decisions before integrating them and that often these decisions are made without fully understanding the business implications which you know can lead to mistakes and.
These mistakes of course decrease the uh confidence that our consumers have in US so i’ just like to ask you quickly I know many of you do Design Systems here I would like you to raise your hand if you justify your decisions with evidence that’s good there’s a solid 40 hands raised this room is 3,000 plus people all right um so we have this.
Issue as well where I used to work at back market so for those who don’t know us we’re a retail website we sell refurbished electronics and like most retailers uh we have product teams who pay very close attention to our conversion rates so how many people end up buying something when they go on a website um so it’s very customary for.
Our colleagues and feature teams to test their ideas and with us being a design system team well of course we don’t test anything you know we just trust ourselves we’re experts right so yeah I think I think this is something that we should address and you know how it goes in Design Systems we.
Often have to advocate for something before doing it we often have to justify why we need budget so I’ll first give you a few pointers as to why we might want to bring evidence driven design and I’ll focus on AB testing because that’s what we had in hand at back market and I think it makes sense right so the most important benefit of AB testing for.
Design Systems is it makes us accountable and we are representatives of our designers our developers our colleagues and it’s important for us to make ourselves accountable so that they can trust in what we’re doing um what I like about AB testing in this context is that we are not the people who Define the metrics the metrics for Success the.
Things that mean the business is healthy that defined by our product leaders and if we can prove that the decisions we make support the business needs then our leaders will be happy with us which is good um and of course we want our contributors and our consumers to see that we hold ourselves accountable able because that means they will trust.
Us um another aspect I would like to highlight is when you give your contributors the means to test their ideas and to see if they are worth bringing forward then you can reduce um a lot of the work that you have to do um you know debating with them um discussing the benefits of um a proposal that they’re making and I think this can.
Be good in very large Design Systems when you have a very wide adoption you can end up uh suffering from the cost of rituals especially contribution rituals so this is something that can be helpful for you can give you back some free time and of course we know what design systems are about quality and consistency um very often when a team.
Works on a on a feature on a proposal when they want to bring forward a change to a design system component they are focused on one user flow they are focused on one product on one page on your website and they are talking about bringing a change to a compon component that is used by many user flows by many products depending on your organization.
And it’s important that we test the outcome of a change in other contexts as well right and this can help you detect when a proposed change from one feature team is only good for their context but not for the others it can also help you understand when you might be interested in introducing variants in your component so it’s very useful if we can.
Get tests done through the design system as well and finally last last but not least um the reason why we at B market wanted to invest in AB testing for our design system was that we wanted to have solid evidence that improving the accessibility of our product was good for us as a business so this is why we engaged in this journey and as we did we.
Realized that it’s kind of difficult so I’m going to walk you through an example of how we could Implement an AB test we’ll have a little example to go through and then I’ll kind of um try in the next 15 minutes to give you a recipe to build your own AB testing uh solution for Design Systems right so product page cool product um back Market’s brand uh.
Uses a black primary color so we have an outout button on this product page which is fully black and it really stands out now the issue with this is that when we um when we navigate with a keyboard on a web browser the focus Outline by default is black as well so it’s kind of difficult to see and we could probably make it.
Better and one of the things that you know we could want to test is what happens if instead of having a black button with a black Focus outline we made it blue with a little gradient see if it stands out a bit more would that help people who navigate with a keyboard so the way we would do this typically this is kind of like the naive.
Implementation and this is what we did originally um we would have a human visit our product page and inside our web app a lot of things would happen so first the app would decide to kind of give an identity to our user who’s visiting so you know if they come back tomorrow on the website in the same browser we want to make sure that we.
Keep them in the same AB test group so we want to have an identity for them and then we’ll ask a feature management tool like Cloud B’s for instance to assign our little user identity to either group a or group b right for any test that we might be running we need to decide in which group we’ll be and then we can start creating our product page and as.
We create a product page I want to bring your attention here to the fact that to run our test we are going to modify the code at that page we’re going to change the code and we’re going to write well okay let’s see so we’re importing all our code um if you’re in group a then I’m going to give you this original button from the design system and then.
If you’re in Group B I’m going to give you some overcode and very often that over code is written specifically in the context of a test sometimes directly in the page and then it’s thrown away at the end and the issue with that is that it turns out that we have thousands of buttons of course across probably 100 page so we.
Don’t want to modify 100 Pages manually that’s just too much work and I’m very lazy so I don’t want to do it um and even even more critically if we if we don’t edit all those pages to make the test across the entire product um we might actually skew the result of a test so here let’s imagine that um changing this Focus outline to make it more.
Visible let’s imagine that it’s good let’s imagine that it works and it helps people add the product to the cart and then move on to the next page well when they reach the next page this keyboard navigation Improvement that we’ve made is no longer available so they might fail to place the order and we might look at the test metrics and think that.
We have not increased conversion rate so we might have a really really good feature that is useful and we might not notice it in our AB tests if we don’t do them through other product so not great then of course another problem we have is our button is used inside our dialogue in the design system and inside many other components so we need to make.
Sure that we modify these as well to make the experience coherent so we’re going to start with a very simple idea we want to make sure that AB testing a design system component does not involve any change in the code right so something needs to happen we import the the button in our product.
Page and we want to receive the right code that we need to run for our our consumer and we don’t know what’s going to happen yet but we’re going to work for it so um very simply when we build a component Library when we build a web app it’s kind of the same thing we have many different source files and many of these.
Files they will reference other files they will import code from other files so import components for instance and we’re going to take all of this files together and all the dependencies They import and we’re going to bundle them into a single file which we will then give to a web browser so our design system might be made of I don’t know 30.
Something components and they will all be in this single JS file that we give to our users the first thing we’re going to do is we’re going to split it so um pretty much every front-end tooling uh available allows you to split your output so we’re going to make one file for every component that we have so that.
We can change one individual file later on and I want to bring your attention here to the fact that that our dialogue uses our button and the dialog.js file which we’re producing will have a copy of a button that is not guaranteed to be the actual button that is in button.js so we still need to do one more thing we um we will use another feature.
Called externals where we would basically declare that we don’t want these dependencies resolved immediately we are going to produce an incomplete build on purpose on the library side and we’re going to tell the web have that it needs to figure it out on its own and it needs to connect the dots so our dialogue that JS will just say I’m.
Missing a button and later on the web app will add the button by finding button.js and this will allow us to do some cool stuff so in the web app in our product page.js we will have direct and indirect references to a button through a variety of components and what we’re going to be able to do now is we’re going to be able.
To replace them through another feature called import aliasing so talk to your front- end Engineers I’m sure they will they will know about this they will be able to help you um we are going to replace the design system button throughout the entire web app because we have managed to make sure that we isolated it from everything else this is.
A key idea and now that we can do that we’re going to replace it with a proxy component that will do the work of the ab test so this is completely dependent on your text stack on your implementation I cannot give you the full recipe there’s still some cooking left to do but very simply we will have this.
Component that Imports the A and the B button it’s asks if we should be in group A or B and then it will use the a button or the B button but what’s nice is that this is no longer happening in the page code this is something that our web app tooling is figuring out and it’s either returning the normal design system button when there’s no test that.
Is going on or if there is a test that is going on it’s going to return that proxy component I just showed you which will figure everything out at the right moment so at this stage I can say we can ab test a design system component we still have a few things to to talk about we still need to discuss when when is our a test when do.
I need to bring a proxy and for which component and how do I distribute the code that I want to use for my design System test so I don’t have enough time to go into every detail so we’ve written an article which you know you can click on the QR code you can visit the URL and you will have the um content available with we’re going to continue to improve.
It and feel free to get in touch with me if there’s something that’s missing there so for people who cannot see the screen the URL is tab. config I 2024 I AB test in lowercase right so I’ll start with how we distribute the design system code normally our contributors the way that they interact with us when and you know.
Their Engineers when they want to propose something they just C it and they show us and they’re like hey this is better um so they used to just opening a poll request and then coming to talk to us so we wanted to keep that process because it’s convenient for them and it’s powerful um so what we did is we brought a lot of automations around.
Our GitHub um repo so that we could just add a label on a PO request and that would automatically take the code in that pull request and build a package of a design system with it and that package we like we made sure that it’s available to the web app later on I’ll show you in a moment um this is very useful for us because when a test is successful when.
We know that it passes the metric that we’re testing against we can just merge the pool request and it’s available we don’t need to write thr away code and we don’t need to write code a second time and we don’t need to go through more processes if it passes a test we can merge it that is very useful so um what we have left to discuss is.
How do we tell our front end tooling that we want to um that we want to test this specific button and not that one so we brought a little common line utility for our engineers and it’s fairly simple for them to use and it’s also again it’s very similar to what they are used to doing because when they do an AB test in the web app they just go to a file and.
They write that oh yeah so I’m I’m going I’m going to test this specific feature I need this feature flag to be quered I need you to go to cloudbees and ask for this feature flag specifically so they write a file in Json file and they commit it so we wanted to have something that is similar so our Engineers if they want to create an AB test they just run.
That command um we will ask them which code they want to use so the pull request that I’ve showed you just earlier each of them is going to be available here as as long as the pool request is open it’s going to show up in the screen and then we will have our users our Engineers select the components that.
They want to test so in that case for this button Focus outline test we will select button we’ll select button rounded which is our icon button and we could select more if we wanted and then we will uh we will ask which flag we should use which has been defined on Cloud B and what this does is it creates a Json file the same Json.
File I was telling you about that has all the information we need for V tooling to Al as the component um that we need to import and to replace them with proxies and it’s creating the two proxies for the two components we’ve selected so at this stage we’ve closed the loop we know that we can properly isolate every design system component in.
The entire stack but we can properly replace them with something that we can tell which components are being tested and that we can tell where to get the code that we want to test with and we have not touched the application code so what have we done with with this and why did we go through all this trouble it took a couple of months to.
Build to be honest with you it’s uh it’s not completely trivial um well we we had originally a pretty terrible accessibility problem on our product um on the screen we can see that there’s a bunch of product cards and on each of his product cards there is a rating rating component that shows the rating.
That was given by previous customers and originally this rating component was pale yellow on white and I’ve actually made it a much darker yellow than it was in production for the purpose of the stock because you wouldn’t see it um and we wanted to test what happens if we move it to Black right and absolutely not surprise here it improved.
Our conversion our conversion rate people actually could use the information to make purchases so they felt more confident so they did um so we moved on to a slightly more complicated U case now this one what I want to highlight here is that there was a bit of disagreement between different feeder teams on how we should address it.
So originally our input font size was too small it was 14 point it should be at least 16 point um and what that caused is that on some web browsers on mobile when you would start to input your information when you’re ordering um your browser would zoom in directly on the input because it’s so small that this is an accessibility feature that is.
Built in um so that was not great and and we had feature teams who wanted to disable this zoom in future because it was ruining the navigation flow and it was making people abandon their their purchases and we wanted to just increase the font size because we don’t want to give it one accessibility and other teams actually requested accessibility.
Compliance um so we put it to a test we just increased the size and we proved that it increases conversion rate that actually it does not cause people to just you know abandon the purchase because the form is longer so so as soon as we had a test result everybody agreed that it was great to do and we could do it but even more.
Importantly um going back to the first slide of his talk we make a lot of decisions without evidence because we’re experts and we know better and sometimes we make mistakes um so we had another kind of usability improvement we wanted to make to our rating we wanted to have a more fine grained um we wanted to have more granularity in the icons that we’re.
Showing so instead of just having a half star field whenever we’re between zero and one or between four and five we want it to be a bit more accurate and we thought that surely because it’s more transparent because it’s it’s more accurate then people are going to feel that it it’s gives them better information and this is not increased.
CVR this decreased CVR we have no idea why because it’s quantitative research AB testing just you know it just says yay or it says nay and we don’t know why but what’s important here is that that we the design system team we had an intuition we thought that with our usability expertise we knew what to do and it turns out that we were wrong it.
Turns out that U well actually you know um because we had this system in place and because we put it to the test we did not lose revenue and this is what’s truly important the reason why we want to bring evidence in our decision processes is to protect us from ourselves it’s so that we don’t make mistakes and so that.
People can trust in the work that we’re doing so what I’m really hoping now is that people you know the design system practitioners like you will start bringing in evidence it can be AB testing it can be something else it can be user research if you have user researches available but there are clear benefits to it and I really hope that.
You will adopt this approach so we should be two here on the stage and my colleague delin Meier who is an outstanding engineer could not make it here she’s back at home home in front so I would just like you uh to please give her a very warm Round of Applause so she can hear you on the live.
Stream thank you
Config 2024: Improve your design system with confidence through A/B testing (Steve Dodier-Lazaro) In this session, Steve will …
Figma Official Channel,config,improve,your,figma, design, product design, tips, tricks, UI design, ux design, app design, figma design, design for figma, FigJam tutorial, prototyping, collaboration, UX tutorial, Design tips, Figma tutorial, Config, design system, tutorial, config 2024, config 24, config24,