Cryptic Data: Challenges and Rewards in Finding and Using It

Using information technology to make data useful is as old as the Information Age. The difference today is that the volume and variety of  available data has grown enormously. Big data gets almost all of the attention, but there’s also cryptic data. Both are difficult to harness using basic tools and require new technology to help organizations glean actionable information from the large and chaotic mass of data. “Big data” refers to extremely large data sets that may be analyzed computationally to reveal patterns, trends and associations, especially those related to human behavior and interaction. The challenges in dealing with big data include having the computational power that can scale to the processing requirements for the volumes involved; analytical tools to work with the large data sets; and governance necessary to manage the large data sets to ensure that the results of the analysis are accurate and meaningful. But that’s not all organizations have to deal with now. I’ve coined the term “cryptic data” to focus on a different, less well known sort of data challenge that many companies and individuals face.

Cryptic data sets aren’t easy to find or aren’t easily accessed by people who could make use of them. Why “cryptic?” As a scuba diver, I donate time to Reef Check by doing scientific species counts in and around Monterey Bay, Calif. Cryptic organisms are ones that hide out deep in the cracks and crevices of our rocky reefs. Finding and counting them accurately is time-consuming and requires skill. Similarly, it’s difficult to locate, access and collect cryptic data routinely. Because it’s difficult to locate or access routinely, those who have it can gain a competitive advantage over those who don’t. The main reason cryptic data is largely untapped is cost vs. benefits: The time, effort, money and other resources required to manually retrieve it and get it into usable form may be greater than the value of having that information.

By automating the process of routinely collecting information and transforming it into a usable form and format, technology can expand the range of data available by lowering the cost side of the equation. So far, most tools, such as Web crawlers, have been designed to be used by IT professionals. Data integration software, also mainly used by IT departments, helps transform the data collected into a form and format where it can be used by analysts to create mashups or build data tables for analysis to support operational processes. Data integration tools mainly work with internal, structured data and a majority have little or no capability to support data acquisition in the Web. Tools designed for IT professionals are a constraint in making better use of cryptic data because business users are subject matter experts. They have a better idea of the information they need and are in a better position to understand the subtleties and ambiguities in the information they collect. To address this constraint, Web scraping tools (what I call “data drones”) have appeared that are designed for business users. They use a more visual user interface design and hide some of the complexity inherent in the process. They can automate the process of collecting cryptic data and expand the scope and depth of data used for analysis, alerting and decision support.

Cryptic data can be valuable because when collected, aggregated and analyzed, it provides companies and individuals with information and insight that were unavailable. This is particularly true of data sets gathered over time from a source or combination of sources that can reveal trends and relationships that otherwise would be difficult to spot.

Cryptic data can exist within a company’s firewall (typically held in desktop spreadsheets or other files maintained by an individual as well as in “dark” operational data sets), but usually it is somewhere in the Internet cloud. For example, it may be

• Industry data collected by some group that is only available to members
• A composite list of products from gathered from competitors’ websites
• Data contained in footnotes in financial filings that are not collected in tabular form by data aggregators
• Tables of related data assembled through repetitive queries of a free or paid data source (such as patents, real estate ownership or uniform commercial code filings).

Along these lines, our next-generation finance analytics benchmark research shows that companies have limited access to information about markets, industries and- economies.Only 14 percent of participants said they have access all the external data they need. Most (63%) said they can access only some of it, and another 14 percent said they can’t access any such data. In the past, this lack of access was even more common, but the Internet changed that. And this type of external data is worth going after, as it can help organizations build better models, perform deeper analysis or do better in assessing performance, forecasting or gauging threats and opportunities.

Cryptic data poses a different set of challenges than big data. Making big data usable requires the ability to manage large volumes of data. This includes processing large volumes, transforming data sets into usable forms, filtering extraneous data and code data for relevance or reliability, to name some of more common tasks. To be useful big data also requires powerful analytic tools that handle masses of structured and unstructured data and the talent to understand it. By contrast, the challenge of cryptic data lies in identifying and locating useful sources of information and having the ability to collect it efficiently. Both pose difficulties. Whereas making big data useful requires boiling the ocean of data, cryptic data involves collecting samples from widely distributed ponds of data. In the case of cryptic data, automating data collection makes it feasible to assemble a mosaic of data points that improves situational awareness.

Big data typically uses data scientists to tease out meaning from the masses of data (although analytics software vendors have been working on making this process simpler for business users). Cryptic data analysis is built on individual experience and insight. Often, the starting point is a straightforward hypothesis or a question in the mind of a business user. It can stem from the need to periodically access the same pools of data to better understand the current state of markets, competitors, suppliers or customers. Subject matter expertise, an analytical mind and a researcher’s experience are necessary starting capabilities for those analyzing cryptic data. These skills facilitate knowing what data to look for, how to look for it and where to look for it. Although these qualities are essential, they not sufficient. Automating the process of retrieving data from sources in a reliable fashion is a must because, as noted above, the time and expense required to acquire the data manually are greater than its value to the individual or organization.

Almost from the dawn of the Internet, Web robots (or crawlers) have been used to automate the collection of information from Web pages. Search engines, for example, use them to index Internet pages while spammers use them to collect email addresses. These robots are designed and managed by information technology professionals. Automating the process of collecting cryptic data requires software that business people can use. To make accessing cryptic data feasible, they need “data drones” that can be programmed by users with limited training to fetch information from specific Web pages. Tools available from Astera ReportMiner,  Connotate, Datawatch, import.io, Kofax Kapow and Mozenda are great examples on where you can get started for leveraging cryptic data. I recommend that everyone who has to routinely collect information from Internet sites or from internal data stores that are hard to access or who thinks that they could benefit from using cryptic data investigate tools available for collecting it.

Regards,

Robert Kugel – SVP Research

The Office of Finance in 2016

The imperative to transform the finance department to function in a more strategic, forward-looking and action-oriented fashion has been a consistent theme of practitioners, consultants and business journalists for two decades. In all that time, however, most finance and accounting departments have not changed much. In our benchmark research on the Office of Finance, nine out of 10 participants said that it’s important or very important for finance departments totake a strategic role in running their company. The research also shows a significant gap between this objective and how well most departments perform. A large majority (83%) said they perform the core finance functions of accounting, fiscal control, transaction management, financial reporting and internal auditing, but only 41 percent said they play an active role in their company’s management. Even fewer (25%) have implemented a high degree of automation in their core finance functions and actively promote process and analytical excellence.

Despite these findings, we believe that today finance transformation is both necessary and achievable. Practical, affordable technology is available to enhance productivity in order to de-emphasize the department’s “bean counting” role and promote its ability to enhance the performance of the entire corporation. Technology enables Finance to be more proactive and more strategic in providing analyses and methods that enhance its capabilities and improve the performance of the entire corporation. Of course, technology by itself will not transform a finance organization, but most of the longstanding issues that it must address to improve performance can be fixed using information technology to address interrelated people, process and data issues in a comprehensive fashion.

Our Office of Finance research agenda for 2016 emphasizes three broad technology-related themes serving the goal of finance transformation:

• Applying a continuous accounting approach to promote greater departmental efficiency and effectiveness
• Adopting technology that promotes action-oriented continuous planning, using rapid, short planning cycles to promote agility, coordination and accountability
• Using software and other information technologies to achieve continuous optimization to promote ongoing organizational alignment across departments and business units.

Continuous Accounting

We introduced the term “continuous accounting” last year to identify the three areas where our research consistently finds tactical roadblocks to achieving a more strategic finance organization. By focusing on these three areas, finance executives can achieve steady gains in effectiveness.

The first area concerns how the organization uses technology and manages information. To enhance effectiveness, finance departments must use software to automate all mechanical, repetitive accounting processes in a continuous, end-to-end fashion. Automation improves efficiency by eliminating the need to have people perform repetitive tasks. For example, we find that most (71%) companies that automate substantially all of their financial close complete it within six business days of the end of the quarter, compared to 43 percent that automate some of the process and just 23 percent that have automated little or none of it. Using software enables the department to manage the flow of data through its processes in a continuous, end-to-end fashion. This ensures data integrity, which in turn eliminates the need for checks and reconciliations that can consume time that could be spent more productively. Data integrity is undermined every time data is re-entered manually or when a spreadsheet is used in a process: for example, when data from one system is manually transferred to another; when the same information is entered twice in two different systems; or when a spreadsheet is used to perform an allocation or a set of calculations.

The second aspect of continuous accounting involves optimizing scheduling of tasks. Continuous accounting incorporates a process management approach that, wherever possible, distributes workloads continuously to flatten spikes of activities, whether in the month, quarter, half-year or year. This approach eliminates bottlenecks and optimizes when tasks are executed. It reduces stress on the department and can eliminate the need for temporary help and its associated expense. Much of the traditional accounting cycle and related departmental practices are artifacts of paper-based bookkeeping systems. These methods dictated the need to wait until the end the month, quarter or year to take accountants off line to perform aggregations, allocations, checks and reconciliations; that rhythm represented the best trade-off of efficiency and control in such antiquated approaches. Today’s systems offer far more flexibility that enables departments to spread workloads more evenly over time and complete them more expeditiously.

The third aspect of continuous accounting is the need to instill continuous improvement in the departmental culture. This steps counters tendency of any organization – but especially finance – to embrace a “we’ve always done it this way” mindset that resists needed change.  Continuous improvement acts as a mission statement that sets increasingly rigorous objectives. To achieve those objectives it’s necessary to have regular reviews of performance toward those objectives and make addressing shortcomings a priority. For departmental executives, communicating the need for continuous improvement is an essential element to achieving finance transformation.

Used as an organizing principle for the department, continuous accounting frees up time and therefore the resources needed to implement changes that result in performance improvements in a sustained and steady fashion. Adopting a continuous accounting approach enables CFOs and finance executives to reduce the amount of time spent “fighting fires,” many of which are the result of not using capable technology.

The Transformation of ERP

In most companies, ERP systems are the backbone of the accounting function, and this software category will continue to be an important focus of our research in 2016. The ERP software market is set to undergo a significant transformation over the next five years. At the heart of this transformation is the decade-long evolution of a set of technologies that enable a major shift in the design of these systems – and it amounts to the most significant change since the introduction of client/server technology in the 1990s. Vendors are seizing on technologies such as in-memory computing, improving the user interface and user experience, adding more in-context collaboration and extending the use of mobility to differentiate their applications from rivals. Those with software-as-a-service (SaaS) subscription offerings are investing to make their software suitable for a broader variety of users in multitenant clouds. These and other topics will be addressed in the results of our next-generation ERP benchmark research, which we will release in 2016.

We’ll also continue to look at the application of financial performance management (FPM) to improve results. Ventana Research defines FPM as the process of addressing the often overlapping issues that affect how well finance organizations support the activities and strategic objectives of their companies and manage their own operations. FPM deals with the full cycle of the finance department’s functions, including corporate and strategic finance, planning, budgeting, forecasting, analysis, closing reporting and statutory filing. In each of these areas, using inappropriate technology has a negative impact on how well a company performs. We will continue to highlight the importance of improving the creation and use of analytics. For example, our Office of Finance research finds that on average, companies that are heavy users of spreadsheets in their closing process take longer to close their books than those that limit them or don’t use them at all. Elsewhere, our next-generation finance analytics research finds a high degree of dissatisfaction with finance analytics in the company: 58 percent said that significant or major changes are necessary while just 7 percent stated no improvements are necessary. The research also shows that heavy use of spreadsheets for all forms of analysis is at the heart of this dissatisfaction. In 2016 also we’ll publish the next installment of our Financial Performance Management Value Index, which assesses vendors and products in this software market.

Financial Performance Management

As noted above, we recommend that finance organizations that want to play a more strategic role in the management of their corporation should adopt a continuous planning methodology for their financial planning and analysis function. A continuous planning approach uses frequent, short planning cycles to promote agility, coordination and accountability in operations. It includes establishing an ongoing dialogue among finance and line-of-business managers and executives to track current conditions as well as changes in objectives and priorities driven by markets and the business climate. To manage planning in such a comprehensive way requires dedicated software that enables members of the FP&A organization to focus more of their time on analysis and modeling. Technology also enhances the quality of plans, forecasts and budgets. In particular, in-memory computing makes it feasible to rapidly process computation of even complex models with large data sets. Consequently, it can expand the range of planning, budgeting, forecasting and reviewing performed in rapid cycles. It enables organizations to run more simulations to understand trade-offs and the consequences of specific events, as well as change the focus of reviews from what just happened to what to do next. For these reasons, in-memory computing also may encourage more companies to replace spreadsheets (which have practical limits to the size, complexity and adaptability of the models that are created in them) with dedicated planning applications that can harness the power of in-memory processing.

Sales and Operations Planning for Finance

Companies that deal in physical goods that are manufactured or sourced and then sold direct or into distribution channels often benefit from using sales and operations planning (S&OP). The process of orchestrating the flow of parts and materials through the production process to meet expected customer demand involves many functional units, each of which make plans, as well as the finance organization, which assesses the financial impact. Sales and operations planning is a discipline aimed at aligning and optimizing the plans of several business units. There are sales plans, product plans, demand plans and supply chain plans. Within a corporation, the performance of each of the functional units that produce these plans is assessed using different, often conflicting metrics. Information technology enables corporations to manage their inventories more skillfully and  minimize their working capital investment while maximizing their ability to fulfill demand. S&OP is designed to align a company strategically so that it can execute tactically in more effective fashion. The ultimate goal is to determine how best to manage company resources, especially inventory and cash, to be able to profitably satisfy customer demand with the lowest incidence of stock-outs. The output of an S&OP group is a SKU-level demand forecast that is used to create a detailed inventory plan. This quantitative plan is a major driver of a process that guides the purchasing an optimal amount of inventory (the one that best balances desired fulfillment rates while minimizing the investment in inventory) from the best set of suppliers (balancing a range of considerations including goods availability, pricing, discounts, economic order quantities and supply chain constraints). To enhance their strategic value, the financial planning and analysis group should play an integral role in the sales and operations planning process.

Advanced Analytics

We also will monitor the ongoing development of advanced analytics for business users. Using technology to make better use of data through advanced analytics can provide companies with breakthrough results. Often that’s because using capable information technology can provide insights and visibility that are unavailable by eyeballing data or using spreadsheets. Advanced techniques such as predictive analytics provide companies with more nuanced forecasts as well as the ability to spot deviations from expected results and thus address problems or seize opportunities sooner. For example, price and revenue optimization is rapidly developing applied analytic techniques that enable businesses to achieve higher profitability, increased sales or some combination. Software that helps manage pricing and profitability is spreading from hospitality, transportation, retailing to consumer financial services and other areas, especially business-to-business verticals. Used properly, this type of software enables a company to tailor its control of individual decisions regarding pricing, discounts and other terms to achieve the results best suited to its strategy. It can continuously make adjustments consistent with longer-term objectives in response to market conditions. Price and revenue optimization is impossible to achieve without using software and analytics that can deal with the huge volumes of today’s data.

Tools for Promoting Productivity and Effectiveness

There are a range of specialized software tools also can promote a more effective finance function, and executives must focus on acquiring and using those that enable the department to take a more active role in improving performance in the company’s operations. Finance has the necessary analytical talent and is positioned to be a neutral party in balancing the requirements of different functional groups or where issues cross business units or geographic boundaries.

The Office of Finance practice will continue to focus on software categories that can improve corporate efficiency, increase visibility and enhance agility. Our main objective is to enable finance organizations to be more effective by eliminating the root causes of time-wasting, low-value activities. For example, more companies are adopting a subscription or recurring revenue business model. This model isn’t always handled well by ERP systems, especially if a company is selling something more complex than simple subscriptions. These companies need to automate their quote-to-cash process from end to end, with the objective of controlling the flow of data, from configuring, quoting and pricing all the way to billing. Using this type of automation to ensure data quality enables companies to achieve two usually conflicting goals: substantially reducing finance and accounting department workloads while still allowing sales and marketing to offer customers flexibility in how they buy their services or products. Expense management is another classic time-waster poorly executed in most companies. Automation not only can save the finance department time, it also can reduce the “administrivia” workload for employees who have to submit expense reports. The cost of these expense management systems is typically less than one full-time equivalent employee, but it can save a multiple of that amount of time.

Managing Taxes More Intelligently

Taxes are one of the biggest expenses corporations face. There are two basic types of taxes: direct or income taxes and indirect taxes, which include sales and use tax and value taxes. Managing direct tax provision and analysis is still in the dark ages in most companies. We recommend to corporations that operate in multiple countries and that have even a moderately complex legal entity structure that they consider tax provision software that is supported by what we call a tax data warehouse of record. Taxes operate in a parallel universe from business management. Our research confirms that most companies use spreadsheets to manage their tax provision and analysis: Half (52%) rely solely on spreadsheets, and another 38 percent mainly use them. Several issues arise in using spreadsheets in the tax function: They are time-consuming, provide limited visibility to senior executives and pose unnecessary risks through errors. International companies are facing increasing scrutiny of their tax positions and can benefit from using dedicated software to manage their direct taxes more intelligently. Among the indirect taxes, in the United States, sales taxes are notoriously complex to administer. We recommend that any company with 100 or more employees doing business in more than a handful of states adopt a sales tax service for the same reason that they use a payroll service: It’s not worth the time, hassle and potential liability to do it in house.

The Impact of Changes to Accounting Rules

The Office of Finance practice at Ventana invests a great deal of time in researching software applications and related information technology. Uniquely, though, we also read accounting bulletins. The world of accounting is undergoing a substantial change now and over the next three years as a result of the adoption of accounting rule changes for revenue recognition and, to a lesser extent, lease accounting. The impact of revenue recognition changes will be profound because it is built on a fundamentally different conceptual framework than classical accounting. The upshot of this framework is that systems must account for revenues and expenses in a parallel fashion rather than in a balancing one. This type of approach would have been extremely problematic in paper-based systems. It’s feasible only because of the nearly universal use of computer-based accounting systems. Almost all ERP vendors are gearing up to support the new accounting rules, but it’s important for companies to plan ahead to make the transition as smooth as possible. And it’s important to be sure that sales contracts and documentation are designed to make accounting for them as efficient as possible.

Technology’s Role in the Office of Finance

One major reason for investing in technology is to help senior executives achieve better results by supporting more effective business management techniques. For example, our benchmark research on long-range planning demonstrates that better management of technology and information can improve alignment between strategy and execution. And when it comes to cloud computing, far from simply being a technology concern, cloud computing enables corporations to cut costs and gain access to more sophisticated technology than they could feasibly support in an on-premises deployment. Using technology can boost performance. The improper use of spreadsheets as seen in our research continues be an unseen killer of corporate productivity because these tools have inherent defects that significantly reduce users’ efficiency. Relying on spreadsheets makes it impossible to find the time to improve performance. Increasingly companies have inexpensive options that are easier to use and enable more advanced, reliable modeling, analysis and reporting.

Information technology is an essential element of business management and promotes a discipline of continuous optimization, a term we use to emphasize the importance of achieving better alignment of organizations to a company’s strategy. Yet many senior executives and managers have too narrow and too limited an understanding of IT’s full potential, much as those managing corporate information technology usually don’t appreciate business issues and how IT can address them. The business/IT divide is a barrier that prevents many companies from achieving their performance potential. The divide need not exist. Business executives don’t have to be able to write Java code or master the intricacies of an ERP or sales compensation application. However, they should master the basics of IT just as they must understand the fundamentals of corporate finance, the production process and – at least at a high level – the technologies that support that process. Our research practice addresses the significant business issues where technology plays an important role in addressing those issues. Because business is dynamic, optimization must be continuous to adapt to changes in markets, the competitive landscape and customer demands. Continuous optimization requires companies to operate in faster cycles and have real-time visibility to improve responsiveness and agility. Information technology can remove the barriers that prevent them from achieving more optimal results.

Regards,

Robert Kugel – SVP Research