Using the Internet as a Source for Information Gathering in our Generation
By Duncan Filshie, Jeremy Gaudet, Neeta Grover, Elsa Koutsavakis
- 1 Background
- 2 Issue #1: Validity of Information from the Internet
- 3 Issue #2: Information Overload
- 4 Issue #3: Ethical Implications of Using the Internet
- 5 How CI Professionals Can Overcome the Issues
- 6 Conclusion
- 7 References
The internet is a worldwide system of interconnected computer networks: it is a network of networks that uses a common data transmission protocol to facilitate data transmission and exchange. The internet facilitates communication of all types by providing a means for data to travel from any connected computer to any other connected computer. For the competitive intelligence professional the internet provides access to millions of terabytes of information that can be used in a variety of ways to produce intelligence. Early in the 21st century the internet provided access to numerous types of information, including: 
- searching for information through search engines
- obtaining knowledge about customers through web analytics
- receiving feedback from customers about products or competitors products
- monitoring discussion groups
- conducting patent searches
- improving stock decisions
- accessing the latest and most current news
- learning about competitors, partners, suppliers and customers by visiting their websites
- searching for and contacting experts
- accessing governmental information and files
- using commercial online databases as information sources
The vast amount of information accessible through the internet is astounding. The internet is an information-rich resource and an inter-organizational communications tool  that significantly altered the way that businesses gather, produce and share competitive intelligence.
From 2000 to 2008, worldwide internet use increased from 6.79% of the world population to 23.9% . Over the same period, internet use in Canada and the United States increased from 43% of the population to over 75% . This substantial increase in internet use has borne a new information source available to the competitive intelligence professional. Termed social media, these information sources, are generally based on the conversations and interactions of people online. These relatively new information sources include:
- Blogs and micro-blogs
- Social networking sites and aggregators
- Social bookmarking
- Social news
- Photography and art sharing
- Video sharing
- Live casting
- Music and audio sharing
- Presentation sharing
- Reviews and opinions
- Product reviews
- Business reviews
- Community Q&A
- Brand monitoring
- Information aggregators
Effective use of these information sources is a challenge that requires a plan for capturing competitive intelligence information and a process for developing a competitive intelligence database  that should address information validity (authority, accuracy, currency, objectivity and scope), information overload and ethical implications.
Issue #1: Validity of Information from the Internet
Using the internet as a source of information can be a risk when it comes to validity. It is well known that the internet is open to any kind of information. Anyone can publish anything they want on the web. There is no approval necessary for publishing on the internet, and so there is a wide variety of quality and value of the information available. For CI professionals of our generation this poses a new challenge of how to validate information found on the web. This section outlines the five criteria for evaluating online information based on Chapter 4 of Competitive Intelligence: A Framework for Web-Based Analysis and Decision Making.
Indicators of authority include credentials, authorship, affiliation and domain information. The authority of a website can often be identified in institutions such as government agencies, non-profit organizations, and educational institutions. In addition, domain information (ie; .com,.edu,.gov) is a good indicator of a reliable site. For example switching the .gov with .com can be the difference between the real site and one full of advertisements. The following questions should be asked concerning authority:
- Who is the author of the site?
- What is the domain type? Does it match other institutions?
- Are there any credentials provided?
- Is the site maintained by a well known organization?
- Is there accurate contact information?
The term accuracy refers to issues such as the verifiability of the information supplied. Errors in grammar, spelling and contradicting information are often clues to a unreliable source. Accuracy can also refer to issues of whether the information is up to date, detailed and comprehensive. For example, old information from a reputable source may be inaccurate. General questions pertaining to accuracy are:
- Is the information verifiable?
- Is the information error-free?
- Is the information timely?
- Is the information comprehensive (ie; does it give the whole picture)?
Currency involves examining problems on a website such as broken links and out of date information. It can be linked to accuracy, but more importantly, how current the website is can help identify if it is legitimate. Questions to ask regarding currency:
- Has the website been updated recently? Is there information on when it was last updated?
- Is some of the information out of date?
- Are there many broken links?
This category refers to information which has a particular point of view (or bias). It is important to be critical of information obtained from websites and bias may be clouding the true information. On the other hand, objectivity can be relevant when a searcher suspects that information is being withheld or manipulated. Many websites tend to link themselves to reputable and credible organizations. This is meant to influence readers and gain confidence in the website. A good way to confirm this is to go to the reputable site to see if they mention the site in question. Questions that can be asked concerning objectivity include:
- Do the facts on the site seem reasonable based on your knowledge?
- Is there any indication of bias?
- Is there an intent to try to sell you something?
- Is the site sponsored by a commercial organization?
Scope is the content of the information and is related to objectivity. A researcher should be interested in whether the topic is fully covered and for whom the content is written for. More important to the evaluation of information is the purpose for which the information was created. Be sure that the intended audience and purpose is appropriate for the type of information you are looking for. Questions to ask concerning scope are:
- Who is the intended audience?
- What is the purpose of the site? Education, general information, or entertainment?
- Are both sides of an issue provided, along with balanced supporting information? Is it comprehensive?
- What is the time period and geographic area covered?
- Are there original documents or only abridged versions?
Apart from the 5 criteria mentioned above, it is important that CI professionals validate all information with an additional source. This rule applies to information found on the internet, as well as other sources. A knowledgeable person is most likely the best person you can use for verification.
Issue #2: Information Overload
The internet is a sea of information. In 2008 Google announced it has indexed 1 trillion unique URLs. For CI professionals this excessive amount of information can result in challenges. The purpose of CI is to not only gather the information, but also analyze it to produce intelligence in which a decision can be made. The problem with having too much information to analyze is that analysis process grows exponentially with the volume of collected data . As a result, information overload can complicate the jobs of CI professionals as they need to make sense of much more. More information also gives the impression to those new to the CI process that today’s CI professionals should produce better intelligence since there is more information available. The reality is that more information does not equate with more knowledge. CI professionals of our generation need to work harder siphoning through a mountain of data in order to focus on finding relevant information. In addition, because the internet is filled with so much data, and there are several tools available to search on the web, it is easy for an inexperienced individual to become overwhelmed and spend an inordinate amount of time gathering data.  The result is a shift in energy spent on data-gathering rather than analysis. Those who do not understand the CI process and spend most of their time gathering data will produce weak intelligence, since the importance lies in the analysis.
CI professionals of our generation face the daunting task of finding order in the chaos of the internet, and so it is important to remember how to find the results we need. The key to effective information gathering is taking advantage of the diverse range of sources and tools.  The chapter, Strategically Searching the Web in Competitive Intelligence: A Framework for Web-Based Analysis and Decision Making gives a valuable overview on how to use the Internet effectively to gather CI information. The following are highlighted ideas:
- When using a search engine (such as Google), searchers can use advanced options such as date range, language, Boolean logic, etc.
- Google’s spider program takes a snapshot of every page and archives it, allowing you to view altered or deleted web pages.
- Enterprise information portals are an alternative option when web searching won’t do because the information needed included subject analysis and dependable sources.
- Metasearch engines are designed to search numerous engines at one time.
- Be aware that search engines have their limitations to the number of URLs indexed, the refresh rate of the engine’s index.
- Be conscious of the "Invisible Web". There is some data which is beyond detection of general search engines. This includes, sites *requiring a subscription, newspaper archive, dynamically created web pages, information found with interactive tools.
- CI researchers should be aware of the tools and limitations of the Web and use judgment and intuition at all times.
- Take advantage of subscription-based inline databases if you can.
- Take advantage of alerting services for coverage of literature. Some journals offer services of emailing you the table of contents whenever a new issue is published.
- Global company information portals (Hoover’s Online, Business.com, etc) are key to assessing the current business environment
- Look at competitor websites as well as their affiliations using reverse link look up on Google.
- Monitor discussion groups, blogs, etc. to see what people are saying about your competitors
Issue #3: Ethical Implications of Using the Internet
Code of Ethics for CI Professionals
The Society of Competitive Intelligence Professionals (SCIP) provides a universal Code of Ethics for CI Professionals:
- To continually strive to increase the recognition and respect of the profession.
- To comply with all applicable laws, domestic and international.
- To accurately disclose all relevant information, including one's identity and organization, prior to all interviews.
- To avoid conflicts of interest in fulfilling one's duties.
- To provide honest and realistic recommendations and conclusions in the execution of one's duties.
- To promote this code of ethics within one's company, with third-party contractors and within the entire profession.
- To faithfully adhere to and abide by one's company policies, objectives and guidelines.
Personal Ethical Codes
In additional to professional or corporate codes, CI specialists are encouraged to commit to a personal ethical code of conduct. An individual’s personal limits must be carefully considered early in any CI activity. Fuld & Company, a leading CI consulting company suggest that people participating in information collection and CI activity ask themselves three key questions:
- How should I represent myself?
- Do I identify my sources in a report?
- Did I "trick" the individual into giving me the information?
Further, the Fuld & Company suggests that to better understand the ethical quality of an person's behavior, individuals should look to the “Harm Rule”: "I will not do anything that may now or in the future harm or embarrass the corporation."
Ethical Codes and the Internet
Most Corporate codes of conduct do not explicitly outline ethical conduct for data collection on the internet. Of the several codes that were explored in Competitive Intelligence Ethics: Navigating the Gray Zone, a small number identified very specific internet-related activities:
- No computer trespassing (obtaining access to a competitor’s network without permission)
- No hacking websites
- No chat rooms
One company specifically outlined a policy for the internet in its Competition Policy and Guidelines but focused mostly on the information being posted or distributed by the company. With the increased use of the internet as a source of data collection, gaps in terms of ethical information research and company counter-intelligence issues will arise.
Ethical Issues with the Collection of Information from the Internet
Common ethical dilemmas in CI
Traditionally, contentious ethical issues have centered around the acquisition of information through
- Bribery or other agreements (information swapping)
- Eavesdropping, stealing or without consented from the provider
Data collection without consent
In the book, “Controversies in competitive intelligence: the enduring issues”, Gregory Windel identifies the danger that many CI specialists believe that “internet-CI is always ethical.”  He reasons that this is not the case since information that one individual views as privileged, could be viewed by another as freely accessible if it can be found on the internet. The only factor preventing an individual from accessing privileged information is the will and personal principles of the professional. Windel argues that internet research requires self-policing, a risky requirement. Individuals and companies that have not provided consent to having CI operations conducted on them may react unfavorably resulting in harm for the professional or his or her company.
Internet–related counter-intelligence issues
Windel’s other concern for internet collection is the requirement to implement counterintelligence measures to protect a company or individual’s own secrets. He argues that these activities raise important ethical questions about posting misinformation.
Misrepresentation over the internet
Individuals can easily misrepresent themselves on the internet to gather information. Misrepresentation includes behaviors such as: 
- Omitting some details about one’s identity
- Not revealing one’s identity after learning classified information without consent from provider
- Not disclosing true intent on how information will be used
Activities to gather information from the internet do not allow CI professionals to properly ensure that they are not misrepresented, therefore do not allow traditional ethics standards of misrepresentation to apply to internet data collection.
Ideas for an Ethics and the Internet Framework
To ensure a sound CI-ethics policy for conducting internet-based CI that guides professionals to carry out research activities within acceptable ethical boundaries, a number of areas should be further explored:
- Misrepresentation on surveys and social media tools (e.g., twitter)
- Company policies regarding the exchange of false information as a counter-intelligence measure
- Use of only public sources (i.e., disallow activities such as hacking)
- Specific rules that apply to the collection of information on sector-specific information, country statistics or business-related information and others that apply to the collection of information on individuals
- Respect for privacy and use of confidential information
How CI Professionals Can Overcome the Issues
The internet is a vast information resource that needs to be used effectively in order to gain the most in a competitive intelligence project. Traditionally, the internet was not a replacement for active techniques involving human sources but the internet could be used to maximize an organization’s competitive intelligence capabilities. 
Although competitive intelligence professionals need to remain aware of the issues surrounding the over-reliance on internet based sources for competitive intelligence projects there is an emerging potential opportunity to use social networks as a means of human source verification of secondary information.
However, competitive intelligence professionals and competitive intelligence projects must still consider the “internet trap” and then must remain aware that social networks provide information and not intelligence.
The Internet Trap
The “internet trap” describes the perceptual blindness that can exist through the over-reliance on internet-based competitive intelligence and involves a number of pitfalls that should be avoided by competitive intelligence professionals, including: 
Internet tools maximize CI efficiency
- Not all publicly available information is accessible through the internet nor do all nations share the same degree of online information infrastructure
- Search engines do not index all internet accessible information (i.e. online databases)
- Information overload
Internet tools minimize CI costs
- The highest quality sources of information is not necessarily available online and those high-quality sources that are available online cost significant amounts of money
- There are substantial costs associated with gathering, disseminating and analyzing online information
Internet-based CI information is reliable
- There is a lack of accountability in online information and information relevancy is an issue
- Data can be stale data and may result in inaccurate competitive intelligence decisions
- Information may be accessible that is posted for counter-intelligence measures
Internet tools maximize user anonymity
- Traffic that accesses websites and information sources on the internet is tracked
Internet-based CI is always ethical
- Information that is viewed as privileged by one individual may be viewed by another as freely accessible if it can be found on the internet
- The use of counterintelligence measures such as posting misinformation
Avoiding the Internet Trap
Fleisher and Blenkhorn developed a framework Figure 1 below, for competitive intelligence projects in order to avoid over-relying on internet accessible information and to avoid the "internet trap”.  Although this framework was developed prior to the predominance of social networks it remains appropriate for choosing the appropriate tool and for developing an internet based competitive intelligence strategy.
Stage 1: Choosing the Appropriate CI Tool
- Identify basic CI information needs of the organization
- Determine if secondary sources of information are appropriate
- Evaluate perceptual blind-spots to determine if internet-based CI is the optimal technique
- Develop an online CI strategy
Stage 2: Develop an Internet-Based CI Strategy
- Conduct an internet CI audit to assess the needs of a CI program and existing organizational capabilities to accomplish CI
- Develop a strategy that avoids the pitfalls highlighted in the internet trap
The framework will ensure that an appropriate CI tool is chosen given the CI products required and will ensure that when an internet-approach to CI is appropriate that the pitfalls associated with information accessed through the internet is avoided. As a result, the quality of CI information obtained through the internet should improve.
The Social Network Trap
With the increase in the number and variety of social media sites and social media users (see http://www.focus.com/images/view/3504/ for a graphical depiction of the boom of social sites), there is an opportunity to use social networks for competitive intelligence activities.
However, social networks, like other internet-based information sources, provide information and not intelligence.  Although social networks can be used as a source of information they, in and of themselves, are not competitive intelligence tools and should not be utilized as such.
It is important that the CI professional recognize that the use of information obtained through social networks suffers from the same issues as information obtained from other internet-sources. Although social networks provide a human element to the information search there are still issues related to validity, reliability and ethics. Social networks only assist in gathering information on the competitive environment and do not support core intelligence processes.
CI professionals of our generation use the internet predominately as their source to gather information. With the overwhelming amount of information, questions of validity, and ethical concerns that are faced with using the internet, it is important to recognize both the limitations and strengths of this powerful tool. Most importantly, CI professionals should have strong CI processes and robust codes of conduct in order to appropriately capitalize on the internet.
- Vriens, Dirk. Information and Communication Technology for Competitive Intelligence. Idea Group Publishing, 2004.
- Thompson S.H. Teo, Wing Yee Choo. Assessing the impact of using the Internet for competitive intelligence. Information and Management, 2001.
- Nordstrom, Richard D. and Richard L. Pinkerton. Taking Advantage of Internet Sources to Build a Competitive Intelligence System. John Wiley & Sons, 1999.
- Vibert, Conor. Competitive Intelligence: A Framework for Web-Based Analysis and Decision Making. Thompson, 2004.
- Fleisher, Craig S. and Blenkhorn, David L. Controversies in Competitive Intelligence: The Enduring Issues. Praeger, 2003.
- Fleisher, Craig S. and Blenkhorn, David L. Managing Frontiers in Competitive Intelligence. Quorum Books, 2001
- Society of Competitive Intelligence Professionals, http://www.scip.org/About/content.cfm?ItemNumber=578&navItemNumber=504.
- Fuld & Company, “A Statement of Ethics: A guide for employees and clients”, 2008, http://www.fuld.com/Company/FuldStatementofEthics.pdf.
- Dale Fehringer and Bonnie Hohhof (Editors). Competitive Intelligence Ethics: Navigating the Gray Zone. Society of Competitive Intelligence Professionals, 2006.
- Trevino, Linda and Gary Weaver. “Ethical Issues in Competitive Intelligence Practice: Consensus, Conflicts, and Challenges, Competitive Intelligence Review”, Vol. 8, No. 1, 1997.