Learning by Shipping: Using data to inform strategy
Aaron Wroblewski stashed this in Software Engineering
This thinking is not only for software - pretty much any business can learn from this. "Story" (thisisstory.com) is a physical store in NYC that opened with a Beta program to test in practice the theory of a physical store that is more agile than an online one.
"The role of data in product development is not without controversy. In today’s world with abundant information from product development teams and analysis of that data, there is ample room to debate and dissect choices. A few common arguments around the use of data include:
- Representation. No data can represent all people using (or who will use) a product. So who was represented in the data?
- Forward or backward looking. When looking at product usage, the data looks at how the product was used but not how it will be used down the road (assuming changes). Is the data justifying the choice or informing the choice?
- Contextual. The data depends on context in which it is collected, so if the user interface is sub-optimal or drives a certain pattern the data does not necessarily represent a valid conclusion. Did the data consider that the collection was itself flawed?
- Counter-intuitive. The data is viewed as counter-intuitive and does not follow the conventional wisdom, so something must be wrong. How could the data overlook what is obvious?
- Causation or correlation. With data you can see an end state, but it is not always clear what caused the end-state. If something is used a lot, crashes a lot, or is avoided there might be many reasons, most not readily apparent or at least open to debate, that cause that end-state. Is the end-state coincident with the some variables or do those variables cause the end-state?
Data transparencyThe use of data is critical to modern product development. Every product of every kind should have mechanisms in place to learn from how the product is used in the real world (note, this is assuming very appropriate policies regarding the collection and usage of this data). This is not just about initial development, but evolution and maturing of the product as well.
If you’re going to use data to design and develop your product, and also talk about how the product was designed and developed, it is worth considering how you bring transparency to the process. Too often, both within an organization and outside, data is used conveniently to support or make a point. Why not consider how you could provide some level of detail that could reduce the obvious ways those that disagree with your approach might better understand things, especially for those that follow along and care deeply. Some ideas:
- Provide absolute numbers for the size of the data set to avoid discussions about sample size.
- Provide a sense of statistical significance across customer types (was the data collected in one country, one type of device, etc.).
- Provide the opportunity for additional follow up discussion or other queries based on dialog.
- Overlay the strategic or social science choices you are making in addition to the data that informed the choices.
Transparency might not remove controversies but might be a useful tool to have an open dialog with those that share your passion for the product."
Putting color, art, making things, and wellness in the midde of the story is pretty brilliant.