In this blog post we present some ideas that one can consider before conducting a qualitative survey to evaluate new software engineering tools and systems. These tips derive from our own surveys conducted on Android developers as part of my research in software engineering.
The back-story
Software engineering is a technical field where researchers and developers try to pinpoint existing problems in software systems and give solutions. Also, they aim to program new software to simplify and automate everyday tasks. Even though there are great ideas related to new software tools, in world venues and journals only a small part of presented tools is finally useful in the industry. It is known that especially research tools are designed to solve specific problems under particular contexts and, thus, they cannot be easily used in practice; reasons include: poor usability, outdated documentation, insufficient portability, difficult building and installation, and so on. In this blog post, we discuss how qualitative surveys’ can be successfully conducted to improve existing software tools. In the following list you might find some abstracted concepts that can help you to craft your own survey.
Sample: find the right subjects
To indicate software tools’ practical impact, researchers conduct qualitative studies (via questionnaires and interviews) on developers. However, for the results of the study to be useful, researchers have to find a good sample of developers and use appropriate techniques to build meaningful surveys. Also, it is well-known that the greater the number of the participants is, more reliable the results are. Here are some practices:
- Personal contacts: Colleagues, friends, and friends of friends would be the majority of the subjects that they are going to carefully answer your survey. So, keep in mind to give them the appropriate details in order to keep the bias as low as it is possible.
- Students: Students (undergraduate and postgraduate) can answer the survey as a part of their academic tasks. They are going to pay more attention on the survey if they have some reward (e.g. a bonus to improve the grade of a course).
- Meet ups: Surprisingly, in our survey we found many developers from developers’ meet ups (e.g. search for teams in meetup.com), where software engineers from startup companies participate in. They kindly accepted to complete our survey. So, it would be a good idea for one to consider approaching such communities of developers.
Delivery method: getting to the people!
In order to approach developers, one could use the following ways:
- Send emails.
- Share the survey via social networks (Facebook, Twitter).
- Upload the survey on the Slack of developer teams (e.g. from meet ups).
- Sharing cards with QR codes of the survey’s site, so that participants can have easy access to the survey.
Tools: how will they fill out your survey?
The following tools can be used for the building a web survey:
- Use forms (e.g. Google Forms).
- Develop your own site for the survey (e.g. using web Frameworks, e.g. Flask).
- Use monitoring devices (e.g. cameras, time-stamps, etc.)
- Use appropriate statistical analysis tools (e.g. R, MATLAB, Python)
Methods: how to scientifically approach your survey
One should carefully consider the following methods to conduct a reliable survey.
- Conduct pilots before the main survey.
- Organize interviews based on bibliography.
- Run an adequate number of experiments.
- Minimize bias using randomization.
- Make available any data (e.g. source code) for the replication of the study.
- Conduct quantitative or qualitative survey based on the type of the research.
Lessons learned
Finally, here are some notes to keep in mind:
- Keep the survey as short as possible (e.g. up to 20 minutes), but without removing important questions or details for the subjects to understand.
- Give specific guidelines to your sample.
- Say before to the subject what they should expect from the survey (i.e. they should answer to close or open questions? They should complete some code? They should evaluate some tools? They are going to be monitored?)
- Write well-structured and clear questions.
- Send kind reminders to people that haven’t answered yet.
- Give some rewards if it is applicable (e.g. small bonus to students, free software access).
- Kindly help the subjects to complete the survey.
PS: Here is our Android survey. If you are an Android or Java developer, think of filling the survey!