Some Problems with Properties: A Study on Property-Based Testing in Industry
Property-based testing (PBT) is a popular testing methodology from the functional programming community that is beginning to see broader use. For testers who already think in terms of formal specifications of the behavior of their code, PBT provides an easy way to check those specifications; for those who do not yet, PBT’s successful track record offers a good reason to start. However, like any up-and-coming technology, PBT has plenty of room to grow and improve. In particular, the challenges and opportunities of applying PBT in an industrial setting are not clearly understood.
We present results from a preliminary study that we conducted to investigate this question. Drawing on interviews with seven professional users of a contemporary PBT tool, we identify several areas sorely in need of attention if PBT tools are to better meet developers where they are. Namely, developers encountered obstacles envisioning properties to test, defining generators, and integrating PBT into their development workflow. We detail participants’ challenges, highlighting directions where further research and tool-building efforts could mitigate the challenges.
We conclude with an agenda for further deepening understanding of the pitfalls and potentials of PBT in industry. We offer a brief overview of research activities we plan to undertake in the coming years, including an expanded interview study to be carried out (tentatively) in partnership with Jane Street, LLC, a large software company that leads in its use of functional programming tools. We hope that advertising this research program within the HATRA community will help galvanize further research into how to make PBT more widely useful.
Wed 7 DecDisplayed time zone: Auckland, Wellington change
09:00 - 10:00
|Program Synthesis Using Example PropagationVirtual
Niek Mulleners Utrecht University, Johan Jeuring Utrecht University, Bastiaan Heeren Open University of the Netherlands, NetherlandsLink to publication
|Some Problems with Properties: A Study on Property-Based Testing in IndustryVirtual
Harrison Goldstein University of Pennsylvania, Joseph W. Cutler University of Pennsylvania, Adam Stein University of Pennsylvania, Benjamin C. Pierce University of Pennsylvania, Andrew Head University of PennsylvaniaLink to publication