Extract Data from Multiple Web Pages into Excel using import.io

In this tutorial, i will show you how to extract data from multiple web pages of a website or blog and save the extracted data into Excel spreadsheet for further processing.There are various methods and tools to do that but I found them complicated and I prefer to use import.io to accomplish the task.Import.io doesn’t require you to have programming skills.The platform is quite powerful,user-friendly with a lot of support online and above all FREE to use.

You can use the online version of their data extraction software or a desktop application.The online version will be covered in this tutorial.

Let us get started.

Step 1:Find a web page you want to extract data from.
You can extract data such as prices, images, authors’ names, addresses,dates etc

Step 2:Enter the URL for that web page into the text box here and click “Extract data”.

Then click  “Extract data” Import.io will transform the web page into data in seconds.Data such as authors,images,posts published dates and posts title will be pulled from the web page as shown in the image below.

Import.io extracted only 40 posts or articles from the first page of the blog!.
If you visit bongo5.com you will notice that the web page is having a total of 600+ pages at the time of writing this article and each page has 40 posts or articles on it as can be shown by the image below.
Next step will show you how to extract data from multiple pages of the web page into excel.

Step 3:Extract Data from Multiple Web Pages into Excel

Using the import.io online tool you can extract data from 20 web pages maximum.Go to the bottom right corner of the import.io online tool page and click “Download CSV” to save the extracted data from those 20 pages into Excel.
Note:Using the import.io desktop application you can extract an unlimited number of pages and pin point only the data you want to extract.Check out this tutorial on how to use the desktop application.
Once you click “Download CSV” the following pop up window will appear.You can specify the number of pages you want to get data from up to a maximum of 20 pages then click “Go!”
You will need to Sign up for a free account to download that data as a CSV, or save it as an API.If you save it as an API you can go back to the API later to extract new data if the web page is updated without the need to repeat the steps we have done so far.Also, you can use the API for integration into other platforms.
Below image shows 20 rows out of 800 rows of data extracted from the 20 pages of the web page.

Conclusion

The online tool doesn’t offer much flexibility than the desktop application.For example, you can not extract more than 20 pages and you can not pin point the type of data you want to extract.For a more advanced tutorial on how to use the desktop application, you can check out this tutorial I created earlier.

Source URL : http://nocodewebscraping.com/extract-multiple-web-pages-data-into-excel/

How to Avoid the Most Common Traps in Web Scraping?

A lot of industries are successfully using web scraping for creating massive data banks of applicable and actionable data which can be used on every day basis for further business interests as well as offer superior services to the customers. However, web scraping does have its own roadblocks and problems.

Using automated scraping, you could face many common problems. The web scraping spiders or programs present a definite picture to their targeted websites. Then, they use this behavior for making out between the human users as well as web scraping spiders. According to those details, a website can employ a certain web scraping traps for stopping your efforts. Here are some of the most common traps:

How Can You Avoid These Traps?

Some measures, which you can use to make sure that you avoid general web scraping traps include:

• Begin with caching pages, which you already have crawled and make sure that you are not required to load them again.
• Find out if any particular website, which you try to scratch has any particular dislikes towards the web scraping tools.
• Handle scraping in moderate phases as well as take the content required.
• Take things slower and do not overflow the website through many parallel requests, which put strain on the resources.
• Try to minimize the weight on every sole website, which you visit to scrape.
• Use a superior web scraping tool that can save and test data, patterns and URLs.
• Use several IP addresses to scrape efforts or taking benefits of VPN services and proxy servers. It will assist to decrease the dangers of having trapped as well as blacklisted through a website.

Source URL :http://www.3idatascraping.com/category/web-data-scraping

Scraping the Royal Society membership list

To a data scientist any data is fair game, from my interest in the history of science I came across the membership records of the Royal Society from 1660 to 2007 which are available as a single PDF file. I’ve scraped the membership list before: the first time around I wrote a C# application which parsed a plain text file which I had made from the original PDF using an online converting service, looking back at the code it is fiendishly complicated and cluttered by boilerplate code required to build a GUI. ScraperWiki includes a pdftoxml function so I thought I’d see if this would make the process of parsing easier, and compare the ScraperWiki experience more widely with my earlier scraper.

The membership list is laid out quite simply, as shown in the image below, each member (or Fellow) record spans two lines with the member name in the left most column on the first line and information on their birth date and the day they died, the class of their Fellowship and their election date on the second line.

Later in the document we find that information on the Presidents of the Royal Society is found on the same line as the Fellow name and that Royal Patrons are formatted a little differently. There are also alias records where the second line points to the primary record for the name on the first line.

pdftoxml converts a PDF into an xml file, wherein each piece of text is located on the page using spatial coordinates, an individual line looks like this:

<text top="243" left="135" width="221" height="14" font="2">Abbot, Charles, 1st Baron Colchester </text>

This makes parsing columnar data straightforward you simply need to select elements with particular values of the “left” attribute. It turns out that the columns are not in exactly the same positions throughout the whole document, which appears to have been constructed by tacking together the membership list A-J with that of K-Z, but this can easily be resolved by accepting a small range of positions for each column.

Attempting to automatically parse all 395 pages of the document reveals some transcription errors: one Fellow was apparently elected on 16th March 197 – a bit of Googling reveals that the real date is 16th March 1978. Another fellow is classed as a “Felllow”, and whilst most of the dates of birth and death are separated by a dash some are separated by an en dash which as far as the code is concerned is something completely different and so on. In my earlier iteration I missed some of these quirks or fixed them by editing the converted text file. These variations suggest that the source document was typed manually rather than being output from a pre-existing database. Since I couldn’t edit the source document I was obliged to code around these quirks.

ScraperWiki helpfully makes putting data into a SQLite database the simplest option for a scraper. My handling of dates in this version of the scraper is a little unsatisfactory: presidential terms are described in terms of a start and end year but are rendered 1st January of those years in the database. Furthermore, in historical documents dates may not be known accurately so someone may have a birth date described as “circa 1782? or “c 1782?, even more vaguely they may be described as having “flourished 1663-1778? or “fl. 1663-1778?. Python’s default datetime module does not capture this subtlety and if it did the database used to store dates would need to support it too to be useful – I’ve addressed this by storing the original life span data as text so that it can be analysed should the need arise. Storing dates as proper dates in the database, rather than text strings means we can query the database using date based queries.

ScraperWiki provides an API to my dataset so that I can query it using SQL, and since it is public anyone else can do this too. So, for example, it’s easy to write queries that tell you the the database contains 8019 Fellows, 56 Presidents, 387 born before 1700, 3657 with no birth date, 2360 with no death date, 204 “flourished”, 450 have birth dates “circa” some year.

I can count the number of classes of fellows:

select distinct class,count(*) from `RoyalSocietyFellows` group by class

Make a table of all of the Presidents of the Royal Society

select * from `RoyalSocietyFellows` where StartPresident not null order by StartPresident desc

…and so on. These illustrations just use the ScraperWiki htmltable export option to display the data as a table but equally I could use similar queries to pull data into a visualisation.

Comparing this to my earlier experience, the benefits of using ScraperWiki are:

•    Nice traceable code to provide a provenance for the dataset;

•    Access to the pdftoxml library;

•    Strong encouragement to “do the right thing” and put the data into a database;

•    Publication of the data;

•    A simple API giving access to the data for reuse by all.

My next target for ScraperWiki may well be the membership lists for the French Academie des Sciences, a task which proved too complex for a simple plain text scraper…

Sources URL :                             http://yellowpagesdatascraping.blogspot.in/2015/06/scraping-royal-society-membership-list.html

Web Data Extraction Services and Data Collection Form Website Pages

For any business market research and surveys plays crucial role in strategic decision making. Web scrapping and data extraction techniques help you find relevant information and data for your business or personal use. Most of the time professionals manually copy-paste data from web pages or download a whole website resulting in waste of time and efforts.

Instead, consider using web scraping techniques that crawls through thousands of website pages to extract specific information and simultaneously save this information into a database, CSV file, XML file or any other custom format for future reference.

Examples of web data extraction process include:
• Spider a government portal, extracting names of citizens for a survey
• Crawl competitor websites for product pricing and feature data
• Use web scraping to download images from a stock photography site for website design

Automated Data Collection
Web scraping also allows you to monitor website data changes over stipulated period and collect these data on a scheduled basis automatically. Automated data collection helps you discover market trends, determine user behavior and predict how data will change in near future.

Examples of automated data collection include:
• Monitor price information for select stocks on hourly basis
• Collect mortgage rates from various financial firms on daily basis
• Check whether reports on constant basis as and when required

Using web data extraction services you can mine any data related to your business objective, download them into a spreadsheet so that they can be analyzed and compared with ease.

In this way you get accurate and quicker results saving hundreds of man-hours and money!

With web data extraction services you can easily fetch product pricing information, sales leads, mailing database, competitors data, profile data and many more on a consistent basis.

Source URL :    http://ezinearticles.com/?Web-Data-Extraction-Services-and-Data-Collection-Form-Website-Pages&id=4860417

Web scraping in under 60 seconds: the magic of import.io

This post was written by Rubén Moya, School of Data fellow in Mexico, and originally posted on Escuela de Datos.

Import.io is a very powerful and easy-to-use tool for data extraction that has the aim of getting data from any website in a structured way.
It is meant for non-programmers that need data (and for programmers who don’t want to overcomplicate their lives).

I almost forgot!! Apart from everything, it is also a free tool (o_O)

The purpose of this post is to teach you how to scrape a website and make a dataset and/or API in under 60 seconds. Are you ready?

It’s very simple. You just have to go to http://magic.import.io; post the URL of the site you want to scrape, and push the “GET DATA” button.
Yes! It is that simple! No plugins, downloads, previous knowledge or registration are necessary. You can do this from any browser; it even
works on tablets and smartphones.

For example: if we want to have a table with the information on all items related to Chewbacca on MercadoLibre (a Latin American version
of eBay), we just need to go to that site and make a search – then copy and paste the link (http://listado.mercadolibre.com.mx/chewbacca)
on Import.io, and push the “GET DATA” button.

You’ll notice that now you have all the information on a table, and all you need to do is remove the columns you don’t need. To do this, just
place the mouse pointer on top of the column you want to delete, and an “X” will appear.

Finally, it’s enough for you to click on “download” to get it in a csv file.
In our example, we have 373 pages with 48 articles each. So this option will be very useful for us.

Good news for those of us who are a bit more technically-oriented! There is a button that says “GET API” and this one is good to, well,
generate an API that will update the data on each request. For this you need to create an account (which is also free of cost).

As you saw, we can scrape any website in under 60 seconds, even if it includes tons of results pages. This truly is magic, no? For more
complex things that require logins, entering subwebs, automatized searches, et cetera, there is downloadable import.io software… But I’ll
explain that in a different post.

Source : http://schoolofdata.org/2014/12/09/web-scraping-in-under-60-seconds-the-magic-of-import-io/

Exploring Web Data Extraction And Its Different Techniques

Web scraping or web data extraction is a distinctive process based on computer software to extract information from different websites. Mostly business organizations are dependent on the web resources for collecting crucial information relating to decision making. With the analysis of such data, they can identify the existing trends of market, details, prices, and product specification. Looking at the time consuming process of manual data extraction, the prominence of data extraction techniques increases.

Different data scraping techniques

Several data extraction techniques are available for the businesses to extract useful information for successful operations. Some of them may include:

    Logical extraction: It comprises logical data extraction of complete source system as well as incremental.
    Physical extraction: This technique involves two different mechanisms for web scrapping that include both online as well as offline.
    HTTP programming: You can also extract data from both dynamic and static websites by implying the technique of socket programming. It allows you to post HTTP requests on the remote web servers.
    Web scraping software: Several software tools are available in the market that serves your individual needs of extracting data with ease. It automatically attempts to recognize the structure of data for a page and extracts the content for further analysis.
    Web scrapping tools: Besides the availability of reliable software, numerous user-friendly web scrapping tools are also helpful in simplifying the entire web scraping process.

Hire a website scrapper

Hiring a suitable website scraper that offers website data extraction services for all your business requirements is an ideal way amongst all other techniques. It provides you filtered and reliable data according to your need for analysis. Some of the major advantages of using website scrapping services may include:

    Automation of data.
    It can retrieve web pages of both static as well as dynamic websites.
    It is also capable of transforming the content into useful information.
    Provides reliable and accurate data.
    It also recognizes several semantic annotations.

Scraping service versus tools

Web scraping services gain more privilege than other tools and software. The basic reason behind this preference is that the service providers are comparatively cheaper than the tools. In fact, they maintain better accuracy and reliability of data.

Summary: It is advisable to look out for suitable web data extraction services instead of any tools or software. This helps in acquiring customized and structured data for your business in legal manner.


 Source : http://www.web-parsing.com/blog/exploring-web-data-extraction-and-its-different-techniques/

Extensive Benefits of Data Mining Services to Marketing – Retail and Outreach Sectors…!!!

There is a vast ocean out there – An ocean of information on internet which is massive, brimming with a lot of data; in fact, it is constantly getting updated, increase the volume with each passing day. In fact, it is believed that around 90% of total information generated in the last two years, is now available on the internet.

Picking right set of information from this heap of data is like searching a needle in the haystack. It is almost next to impossible to search it manually – You need a powerful magnet in form of data mining service provider…!!!

Data mining services work like a magnet – It helps you in finding the right kind of information from huge databases available in the digital world. And with databases getting mammoth every minute, the importance of partnering with a professional and reliable data mining company cannot be overlooked.Though, loaded with a lot of negative connotations; data mining still reigns like a king! In fact, in order to truly appreciate the concept behind data mining, one needs to know it in its entirety.

Every coin has two sides – If there is a brighter side; there tends to be a dark side as well. Though, advantages of web extraction, outweighs disadvantages the fact is it is always the dark underbelly that is highlighted and shown to the world. However, as wise men say, focus on positive sides – Lets see what amazing advantages it can offer to your business and how well you can gain from hiring a professional data mining services.

Upside or Advantage of Data Extraction Services:

While data mining is used primarily in business, it is interesting to know that benefits of data mining goes beyond and across boundaries; it helps various industries as well.

Marketing/Retailing

Data mining can prove to be extremely helpful to the marketers and retailers who are looking out for potential clients as well as aspires to maintain consumer satisfaction. This is one of the methods that allows the businesses to know their potential clients better by acquiring their personal information and preferences.
Not just data extraction helps in determining the trends in goods and services by presenting an overview of online data. With adequate information, you can improve your goods and services, along with changing or choosing the ones which are more in demand. Consequently, success in business has been made quicker and easier these days because of data mining.
Streamline Outreach

Outreach forms an integral part of any business – And to effectively carry out outreach activities; one needs to have a huge cache of database, that can help the marketers to learn how to approach a particular set of customers. Information like that includes relevant e-mail addresses, mailing addresses or social media pages needs to be streamlined any mailers to get the best results.

Data extraction makes this easier; since it gets all the updated information; and in process saves your time and money.

And as it is “the lotus flower grows in mud, but makes our world fragrant” – data mining services is marred by criticism and controversy; however, its extensive advantages outweighs these negativity to a great extent.

Source : http://www.habiledata.com/blog/extensive-benefits-of-data-mining-services-to-marketing-retail-and-outreach-sectors/