How to filter the crawler data using java?

Question

we already fetched the URLs and stored in the db using jsoup lib.Now we are looking to extract the data and store in db,but we are looking only specific field,rather than storing the whole page. for example :http://www.flipkart.com/shoes/ when we fetch this link, we need field like brands ,prices, reviews etc.. using java code!! Please help !

Answer 1

There are two ways you can filter out the whole content,

1. Apply Regex on the response content and extract the needed fields.
2. Using xpath you can extract the needed fields (Preferred and recommended way of parsing).

Ex: 1 - Regex

1. Generate the regex pattern for your selected page.
2. Get the response as String and apply the pattern and retrieve the data.

Ex: 2 - XPath

1. Identify the methodolgy to locate each and every html element uniquely (Or list)
2. Get the response as html/xml form and apply the xpath on the retrieved content and get the data.