Inside the mind of KM measurement

The thinking behind methods of KM and knowledge-perfomance management practiced at international firm Latham & Watkins LLP. By Marcus Willamowski, Latham & Watkins LLP.

Rumour has it that knowledge management (KM) makes sense. But just how much has been a question that knowledge-intensive organisations like law firms have been struggling with for a considerable time.

Of course, there is a plethora of theoretical and empirical analysis on the impact of KM, partially venturing so far as to link its performance to certain financial indicators. Identification and evaluation of those indicators, however, requires long-term projects with appropriate staffing and funding. Those organisations that have subscribed to KM are now capable of reaping the benefits and exploring ways of innovation and marketing of their knowledge, putting them ahead of the competition. Those who have not, and therefore are quite sceptical as to the impact of KM in the first place, are quite likely to be afraid of throwing good money after a lost cause. But would they really be doing so?

Knowledge cannot be measured
It has long been thought that the virtue of KM can be found in its improvement of firm culture, and such that improvement of cultural aspects – for example, the benefits of changing a firm’s culture towards one of knowledge harvesting and sharing – cannot be expressed in dollars or euros, or any other currency. Statistical methods of any kind have been considered to not effectively capture the core aspects of KM. Some basic considerations – for example, measuring access rates to KM databases, have even been sniffed at by some of the theoretical thinkers within KM. Such measurements, they say, touch only the surface of the problem and should be attributed to the area of information management so, therefore, are somebody else’s problem – namely the ICT department. KM is a people’s business and since ICT can merely be a facilitator, if at all, this is proof enough that measurements such as these are not really adding any value. Besides, you cannot measure knowledge, since it is an intangible asset and therefore any KM system should focus on the high-class project of gaining access to and extracting the implicit knowledge in co-workers heads.

You cannot manage what you cannot measure
That you cannot measure knowledge sounded rather plausible in the early days of KM and without any professional enthusiasm for implementing projects based on an abstract idea of organisational theory in the information economy, KM presumably would not have had the impact and success it has finally gained. However, as with any theory, KM must prove that its promises have become reality, if it does not want to end up in the realm of business fads. In fact, the suggestion that knowledge is an implicit, intangible asset does not exclude measuring it in ways that can help the firm evaluate its efficiency. Is measuring knowledge difficult? It can be, but it does not have to be so. Think about the very basic approach to measuring the value of a organisation’s intangible assets by subtracting the value of its tangible assets from its stock market value. Think about Microsoft. But then, law firms do not have a market price or maybe not even a regular balance sheet.

Methods of measurement
Qualitative versus quantitative analysis
KM staff in organisations have experimented with several forms of measurement. Qualitative analysis has a long history within KM, often using questionnaires, feedback interviews and critical-success-factor methods.

Quite on the contrary, quantitative measurements, on the grounds stated above, have been considered to provide limited gain for measuring the performance of KM systems. Such evaluations, however, can help to avoid the drawbacks of qualitative analyses, especially in the subjective judgement of empirical results. Therefore, a quantitative research approach is designed to represent a tangible, visible and comparable ‘ratio’. In other words, quantitative analyses can be used to measure the explicit knowledge of the firm or individuals within the firm, with both financial (for example, basic financial statement analysis, Tobin’s Q, return on investment, net present value) and non-financial indicators – login rate, click-rates, number of documents, number of communities of practice (CoPs). From a KM theory point of view, all methods of quantitative analysis are linked to the fact that any knowledge has to be explicated (in words, writing or other) before it can be transferred to another individual. It is this single moment of explication, which all quantitative analyses relate to, that enables the provision of much more solid data than qualitative analyses.

The Balanced Scorecard
With the advent of the Balanced Scorecard, organisations were equipped with a powerful tool to evaluate the impact of behavioural aspects on overall performance. The underlying concept of the scorecard was that all aspects of measurement have their drawbacks. However, if an organisation offsets some of the drawbacks of one measure with the advantages of another, the net effect can lead to decisions resulting in both short-term profitability and long-term success. As a result, the Balanced Scorecard suggests that pure financial measures be supplemented with additional ones, qualitative and quantitative, reflecting, for example, customer satisfaction, internal business processes and the ability to learn and grow.

Benchmarking (external)
Besides all these internal approaches to measuring knowledge, organisations have also experimented with external measurement systems. External performance-measurement methods are always used to compare against benchmark organisations, primary competitors, or whole industry average. With benchmarking or best-practice methodologies, firms can understand the performance of their KM programmes compared with those of their competitors. Benchmarking is also seen as a tool for identifying, understanding and adopting reasonable practices, in order to increase the operational performance of intellectual capital. From an organisational-learning perspective, benchmarking is concerned with enhancing overall organisational performance, by establishing standards against which processes and products can be compared and consequently improved.

Profiting from derivatives
Price
We find that there is a plethora of models for measuring knowledge performance to pick from, which are widely discussed in scientific journals and proven by empirical research. The downside to most of these models it seems is that they essentially all measure different things, the Balanced Scorecard really being the only exception, by tying different methods of measuring together to provide a broader picture. But a picture of what, exactly? You know the stock market value of your intellectual capital and you know how many experts your firm has on a particular area of law topic – you know the price tags attached to each aspect of KM. When tracking the data over several periods (most likely financial years), one may even be able to provide senior management with trends. But are we really doing a good job in KM, if we have more CoPs this year than we had last year?

Value
With all those tools at hand, the problem today does not really lie in gathering the data that is required to evaluate the benefits of KM, but to relate this data to the specific organisation, its projects, employees and its overall performance. Still, available methods of knowledge-performance measurement can hardly provide us with an estimate of the value of KM. Do they have to? In my opinion, the answer is no. When evaluating the business concepts of some of the more recent successes of disruptive innovation in corporate history, such as Amazon, Wikipedia or Google, it becomes clear that these organisations’ knowledge is not represented by customer-facing products, but by the internal knowledge about customers and the markets they service in general. This is achieved by extrapolating the performance indicators relevant to the organisation from quantitative data. For example, Google measuring the number of hits to a website or keeping track of the user search terms used most often is one thing. Deriving from this merely quantitative data an overall picture of customer behaviour and interests is an entirely another and this, not the know-how required to build a search engine, is exactly the knowledge that makes the organisation powerful. It is this knowledge of individual customer preferences that enables the organisation to stay ahead of its competitors.

Measuring the ‘speed’ of knowledge
Therefore, first measuring the degree of customer or client satisfaction needs should be at the heart of every attempt on measuring KM, regardless of who the client may be, internal or external.

Deriving from derivatives
The problem with ‘best practices’
The best practice approach is an essential component of KM. I, again, have read this statement in many different articles and publications recently, and consider it to be one of the most commonly misinterpreted or even false ideas in KM history. Admittedly, KM should focus on harvesting and distributing all the different kinds of knowledge within an organisation. It is widely understood that one of KM’s more practical aims is to avoid important knowledge becoming lost within the organisation, be it the partner retiring or leaving for another firm. There are two problems with this approach. First, in such an instance, all relevant knowledge in such a persons head would have to be explicated in a short period of time, which is almost as unlikely to be practicable as it is expensive to attempt. Second, the lack of willingness to share knowledge in such circumstances could prevent the firm from accessing the stored information. In a worst-case-scenario – for example, in the absence of long-term projects to substantially embed important individuals’ knowledge throughout the firm using training, coaching and mentoring – the firm would have to start over without the knowledge of the individual in question. Is this bad? Is the firm losing out on its chance to build upon its knowledge? In my opinion, no.

Innovation in risk-averse environments
Junior professionals, especially in knowledge-intensive organisations, tend to be risk-averse. This, and not the juniors’ fondness for ICT gadgets, is the reason why they are the main customers of KM systems. Providing junior professionals with organisational best practices, stored in knowledge databases, helps them find their way in their careers by building on the firms existing know-how. However, it also greatly hinders innovation. There is no explicit reason for junior professionals to deviate from the existing know-how unless an organisation enables them to do so and incentivises such behaviour. In many organisations, this leads to knowledge databases that capture the status quo. Measuring access to such databases, for example, therefore only captures the number of individuals relying on the status quo of the organisation by using its best practices. Some firms therefore have gone so far as to call their explicit knowledge, whatever it be, ‘pretty good practice’.

They have also learnt that measuring knowledge is not restricted to capturing the current impact of KM in general. In fact, if a firm wants to stay ahead of its competitors, it also has to measure the potential of its individuals to adapt to new situations and to generate knowledge and ideas from scratch.

Measuring the ‘acceleration’ of knowledge
Therefore, the firm’s ability to improve and innovate based on its existing knowledge should be the final goal of performance measurement projects and implementations in the KM arena. ?

Marcus Willamowski is a business development and knowledge manager at international firm Latham & Watkins LLP. He can be contacted at marcus.willamowski@lw.com