Used maven for htmlunit dependency for the webscraper. The main issue is that my scraper returns null instead of links. I made an item class to set and get.
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import java.util.List;
public class Scraper {
private static final String searchUrl = "https://sfbay.craigslist.org/search/sss?query=iphone%208&sort=rel";
public static void main(String[] args){
WebClient client = new WebClient();
client.getOptions().setJavaScriptEnabled(false);
client.getOptions().setCssEnabled(false);
client.getOptions().setUseInsecureSSL(true);
HtmlPage page = client.getPage(searchUrl);
List<HtmlElement> items = page.getByXPath("//li[@class='result-row']");
for(HtmlElement htmlItem : items){
HtmlAnchor itemAnchor = ((HtmlAnchor)htmlItem.getFirstByXPath("//a[@class='result-image gallery']")); //itemAnchor gets the anchor specified by class result-image gallery//
Item item = new Item();
String link = itemAnchor.getHrefAttribute(); //link is extracted and initialized in string//
item.setUrl(link);
System.out.println(item.getUrl()); //why don't you work//
}
}
Result: basically a line of null going down
*note: Putting System.out.println(link) returns one link and reuses that same link as it prints new line, in this case it would be just the link ‘https://sfbay.craigslist.org/sby/mob/d/san-jose-iphone-plus-256-gb-black/7482411084.html’ going all the way down
I’m a complete beginner in this cruel world. Any help is useful. edit: I’m going to include the dependency code here just in case, and the code for the Item class likely doesn’t need to be here as it is just a set and a get method outlined by setUrl and getUrl
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.60.0</version>
</dependency>
Advertisement
Answer
This works here
public static void main(String[] args) throws IOException {
String url = "https://sfbay.craigslist.org/search/sss?query=iphone%208&sort=rel";
try (final WebClient webClient = new WebClient()) {
HtmlPage page = webClient.getPage(url);
// webClient.waitForBackgroundJavaScript(10_000);
List<HtmlElement> items = page.getByXPath("//li[@class='result-row']");
for(HtmlElement htmlItem : items){
HtmlAnchor itemAnchor = ((HtmlAnchor)htmlItem.getFirstByXPath("a[@class='result-image gallery']"));
if (itemAnchor != null) {
String link = itemAnchor.getHrefAttribute();
System.out.println("-> " + link);
}
}
}
}
producing something like
-> https://sfbay.craigslist.org/eby/pho/d/walnut-creek-original-new-defender/7470991009.html
-> https://sfbay.craigslist.org/eby/pho/d/walnut-creek-original-new-defender/7471913572.html
-> https://sfbay.craigslist.org/eby/pho/d/walnut-creek-original-new-defender/7471010388.html
.