The report originally came out in July 2013, when Epic was in the process of an AIM listing. Here is a quote from their official Admission Document. “Epic was rated first for ‘‘Potential and Performance’’ by Elearnity in July 2013.” From a glance at the chart above, I suspect that is what most of us would have said if our bespoke e-learning services had appeared in the top right corner. Elearnity retweeted the Epic statement without any qualification.
The 9-Grid™ is a serious and considered piece of research. But how meaningful is the information in the way it is presented? Does it really support Epic’s statement? Or is the medium obscuring the message? As a statistician I take a particular interest in the use and misuse of statistics, so let’s take a closer look. “When I use a word”, Humpty Dumpty said, “it means just what I choose it to mean” My first query is with the terminology. Here is another 9-Grid™, this time for LMS vendors. As you can see, CornerStone is the only vendor in the “Strategic Leader” section, high in both Performance and Potential, while only Cornerstone and NetDimensions are rated as strong performers.
Naturally Cornerstone reported this accolade on their website and on Twitter. Why wouldn’t they?
What does ‘Performance’ mean to you? Perhaps the build quality, range of features or ease of use of the LMS, or some measure of fitness for purpose? In Elearnity’s model, ‘Performance’ is defined as a combined measure of: a) “How often do vendors get shortlisted and how often do they win?” and b) “Corporate customer advocacy” My own view is that (a) is a rather sweeping assumption that the amount of new business chased and won by competitive tender is a valid performance measure. Doesn’t this statement more reflect business strategy and marketing spend, not necessarily performance? Many LMS providers will say only a small proportion of their annual revenue comes from new business won through competitive tender, and some avoid that route altogether. Point (b), I believe, is more appropriate in principle. There should be some correlation between performance and customer satisfaction, but this is more difficult to measure, and the data are unlikely to be statistically valid. The Elearnity model combines these two elements to arrive at a simple overall score of 1, 2, or 3 for Performance. Only Cornerstone and NetDimensions achieve a 3 in the latest published 9-Grid™. Does that make them the best performing LMSs in Europe, or does it suggest they have the biggest marketing budgets? Equally, in terms of investing in bespoke e-learning, many clients might be more interested in a provider’s instructional design skills, subject matter expertise, creativity, and delivery within budget and on time, than in whether they get on lots of shortlists. Potential The other major axis on the 9-Grid™ is ‘Potential’. Again, what does that mean to you? Future growth prospects of the vendor perhaps, or their product development plans? Or is it the potential of the product or service to deliver added value features? For bespoke e-learning, the Elearnity measure of ‘Potential’ is a combination of:
- “The breadth of sector focus such as Finance, Retail, Telecoms, etc.”
- experience “developing for different learning contexts”
- “sophistication” of learning created.
In my experience, most customers are at least as likely to be interested in whether the vendor understands their business and the subject matter, and may well prefer industry expertise to a broad spread across different industry sectors. Welcome to the Fifth Dimension There are another three dimensions to the 9-Grid™. 1) Cost of Ownership - a perfectly good measure and there is no ambiguity in the Elearnity definition. But are the results over-simplified by being represented in a 1 to 3 score? 2) Presence - is a measure of the size of the business, again represented in a 1 to 3 score. Could this be the place to include those elements of the number of tender lists and spread across sectors? 3) Direction of travel – Finally, in a valiant attempt to include five metrics in a two-dimensional Grid, Elearnity provide a measure of “our sense of their likely future direction of travel within the model”, by where they position the vendor’s blob within whatever square of the Grid they find themselves. So if a product is in the top right corner of a box, it doesn’t actually imply they have more ‘potential’ and ‘performance ’than someone in the center or bottom left of the same box. Confused? Me too. Other Dimensions? There are likely to be other factors relevant to any buying decision but which are not covered by the Grid. Take financial stability for example. Kallidus has an excellent record of eight years of consistent profitable growth. By contrast, in a quarterly report to the United States Securities And Exchange Commission (SEC) filed in November 2013, Cornerstone said, “we have a history of losses, and we cannot be certain that we will achieve or sustain profitability,” and that they “expect to continue to incur operating losses as a result of expenses associated with the continued development and expansion of our business.” Do buyers take this sort of vendor information into their considerations? Should the quality of customer service and support also have more prominence in the model? Is this aspect adequately covered as an element of the 1 to 3 score within ‘Performance’. With ‘Potential’ and ‘Performance’ as the two major axes, the natural conclusion is that the best place to be is the top right of the Grid. That is the conclusion that Epic and Cornerstone came to, along with every vendor I personally have discussed the Grid with. Opening up the debate, In conclusion, the 9-Grid™ is a well-intentioned attempt to present the results of Elearnity’s research in an eye-catching visual. They set out “to create a model that is focused on being accessible and understandable to Learning and Talent professionals”. Have they succeeded? What’s your view?
Explore our Access Digital Learning and Compliance software