Evaluation of scheme impact through RCT

Abstract

Between February and March 2021, 60 SMEs based in 7 different EU countries (i.e. Italy, Germany, Finland, Lithuania, Spain, Estonia and Denmark) took part in an online User Experience Challenge (UX Challenge) pivoted on the Design Sprint methodology. To establish if taking part in the UX Challenge increased companies’ digital design readiness and awareness, the access to the UX Challenge was strictly dependent on randomization. This evaluation design allowed comparing a set of indicators of digital design readiness and awareness in the 60 participating companies (i.e., the treatment group) with the same indicators measured in a set of 130 equivalent companies, which served as a "control group". The results of this Randomised Controlled Trial (RCT) suggest that the UX Challenge is a promising way to improve participants’ objective and practical knowledge about design sprint and digital design. Positive, even if not statistically significant, impacts were also found on participants’ positive attitudes towards digital design. However, participants did not show any higher intention to adopt digital design in their companies as compared to the control group. Some organizational and financial constraints may be the reason behind this discrepancy between the largely positive impacts on knowledge (and the tentative positive impacts on attitudes) and the zero impacts on intention to adopt design sprint. More research is needed to investigate these aspects further and to understand whether--and under which circumstances- -the increased knowledge about digital design leads to concrete, tangible changes in companies’ approaches to digital design.