Web Scraping for Investments

Executive Summary

Investment firms - hedge funds, sell-side research groups, traders, and asset managers generally - are increasingly relying on the web as a source of data to improve their investment and trading performance. The motivation is clear: With more than four billion Internet users globally and double-digit zettabytes of data being created annually, the opportunities to gain an investment edge have never been greater.

Opimas estimates that web scraping for investments already accounts for an astounding 5% of all web traffic and that this activity will continue to grow rapidly in coming years.

Table of Contents

EXECUTIVE SUMMARY

 

2

OVERVIEW

 

3

 

THE ULTIMATE DATASET

3

 

WEB SCRAPING FOR INVESTMENTS

4

 

THE WEB-SCRAPING PROCESS

7

USE CASES

 

11

 

SENTIMENT ANALYSIS

11

 

ECONOMIC DATA

11

 

AGRICULTURAL DATA

11

 

COMPANY REPORTS

12

 

MARKET DATA

12

 

TRANSPORTATION DATA

12

 

ENVIRONMENTAL, SOCIAL AND GOVERNANCE DATA

12

 

REAL ESTATE PRICES

12

 

CORPORATE ACTIONS

13

 

REFERENCE DATA

13

 

PRICE COMPARATORS

13

 

RETAILERS AND CONSUMER PRODUCTS

13

 

TRADING AND INVESTMENT DATA

14

SOLUTION PROVIDERS

 

15

 

CRUX INFORMATICS

15

 

IMPORT.IO

15

 

SCRAPINGHUB

16

 

SEQUENTUM

16

POTENTIAL IMPEDIMENTS

 

17

 

REGULATION

17

 

JUDICIAL RULINGS

17

 

ANTI-WEB-SCRAPING DEFENSES

18

LOOKING FORWARD

 

20

© OPIMAS, 2019. All Rights Reserved.