Andreas Stefik on Programming Language Human Factors Research

This interview with Andreas Stefik on empirical studies of human productivity with PL features is fascinating. In short: There aren’t many.

Very few randomized, controlled experiments in the field have ever been performed. This 2015 thesis found only 22!

Much if not most PL human factors research is anecdata. Even the few randomized, controlled experiments which have been performed have never been reproduced.

CS folks who have ever looked down their noses at social sciences for soft research standards should be mortified.

Never take blog posts or tweets claiming: “<study> found <author’s pet PL feature> boosts productivity by <x>” at face value. Conclusive, reproduced results on such a thing would be unprecedented in CS research.