Parsing a large amount of various information.
Daily, weekly, monthly and quarterly multiple parsing a large amount of information on companies and funds whose shares are traded on the US stock market. Demo version of one of the parsing modules and reports in html on downloaded data.
Long-term parsing of a large amount of various information from 4 data sources
Data were downloaded from 4 sources for more than 7,400 companies and funds for more than a year. The data was written to the MySQL database, the volume of which was more than 4 gigabytes. For all these companies and funds, data on stock prices and sales volumes were downloaded weekly (with the possibility of daily download, which was not carried out to save time and for the need for these data every day). Stock data for more than 700 companies and funds were downloaded from nasdaq.com, for them, from the same source daily (that is, a few hours after the end of each day of trading) downloaded data on options (hereinafter - options). Data on shares for other companies and funds were downloaded from finance.yahoo.com Also, after a while, constantly downloaded data on options more than 4.2 thousand companies with finance.yahoo.com. Every 4 weeks a very large volume of various financial and analytical data on all these companies and funds was downloaded from 3 sources: nasdaq.com, finance.yahoo.com and zacks.com (this was called "passports" of companies and funds). Initially, all data are downloaded ordinary personal computer, the data on the "passports" are downloaded via a physical dedicated server, networked with this computer, including using public proxy servers (zacks.com banned my ip address, although the injection from him and was really huge, and the more frequent). Once a quarter for all companies and funds downloading data on quarterly surprises from nasdaq.com. One time with gurufocus.com/financials a large amount of data was downloaded for all companies for large periods of time (up to 10 years, including the last 5 quarters) on their financial reports.
Below are screenshots with explanations that demonstrate the operation of one of the parsing modules. Learn more about the work stated in the tooltip to it. The module and the help to it are made in Russian.
The appearance of the module when you open it:
The module has many settings:
As well as the viewers and editing data downloaded and recorded in MySQL:
This downloaded data can be easily compare with the original by clicking on the buttons to display data on shares and options in data source:
You can quickly and easily add new ticker (abbreviated name of company or Fund on the stock exchange) as manually, and from the list of previously downloaded tickers:
Stock data are downloaded from 2 sources (NASDAQ and Yahoo Finance). The following few screenshots illustrate the parsing of this data from these sources and viewing them:
Parsing the quarterly surprises:
Final screenshots parsing of data on options and their comparison with data in the primary source are illustrated. It also illustrates the re-download of data if two consecutive attempts are made file downloads were not successful:
The following video shows some part of the functionality of the application: