文章详情

短信预约-IT技能 免费直播动态提醒

请输入下面的图形验证码

提交验证

短信预约提醒成功

Java爬虫实现Jsoup利用dom方法遍历Document对象

2024-04-02 19:55

关注

先给出网页地址:

https://wall.alphacoders.com/featured.php?lang=Chinese

主要步骤:

利用Jsoup的connect方法获取Document对象

String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();

内容过长,就不再显示。

我们以这部分为例:

<ul class="nav nav-pills"> 
    <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow"  rel="external nofollow"  >About Us</a></li> 
    <li><a href="https://alphacoders.com/site/faq" rel="external nofollow"  rel="external nofollow"  >FAQ</a></li> 
    <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow"  rel="external nofollow"  >Privacy Policy</a></li> 
    <li><a href="https://alphacoders.com/site/tos" rel="external nofollow"  rel="external nofollow"  >Terms Of Service</a></li> 
    <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow"  rel="external nofollow"  >Acceptable Use</a></li> 
    <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow"  rel="external nofollow"  >Etiquette</a></li> 
    <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow"  rel="external nofollow"  >Advertise With Us</a></li> 
    <li><a id="change_consent">Change Consent</a></li> 
</ul> 

我们先找到所有的ul:

Elements elements = doc.getElementsByTag("ul");

输出如下:

<ul class="nav navbar-nav center"> 
 <li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li> 
 <li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美奖品</a> </li> 
</ul>
<ul class="nav navbar-nav navbar-right center"> 
 <li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="/file/upload/202211/13/q3sz2cejppk.jpg" alt="Chinese-flag">   中文    </a> </li> 
 <li> <a  href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登录</a> </li> 
 <li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 注册</a> </li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a id="prev_page" href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li class="active"><a>1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=8" rel="external nofollow" >8</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=9" rel="external nofollow" >9</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=10" rel="external nofollow" >10</a></li> 
 <li><a>...</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >319</a></li> 
 <li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li class="active"><a>1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
 <li><a>...</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >319</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt;&lt; </a></li> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
 <li><a title="末页 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" > &gt;&gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
</ul>
<ul class="nav nav-pills"> 
 <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow"  rel="external nofollow"  >About Us</a></li> 
 <li><a href="https://alphacoders.com/site/faq" rel="external nofollow"  rel="external nofollow"  >FAQ</a></li> 
 <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow"  rel="external nofollow"  >Privacy Policy</a></li> 
 <li><a href="https://alphacoders.com/site/tos" rel="external nofollow"  rel="external nofollow"  >Terms Of Service</a></li> 
 <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow"  rel="external nofollow"  >Acceptable Use</a></li> 
 <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow"  rel="external nofollow"  >Etiquette</a></li> 
 <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow"  rel="external nofollow"  >Advertise With Us</a></li> 
 <li><a id="change_consent">Change Consent</a></li> 
</ul>

可以发现class为"nav nav-pills"的只有一个,我们找到它:

Elements elements = doc.getElementsByTag("ul");
        //System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
    if (element.className().equals("nav nav-pills")) {
        tempElement = element;
        //System.out.println(element.className());
        break;
    }
}

循环遍历这个ul,输出其中每一个li里每一个a的href和rel属性:

Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
    Elements element2 = element.getElementsByTag("a");
    for(Element element3 : element2) {
        String hrefString = element3.attr("href");
        String relString = element3.attr("rel");
        if(hrefString != "" && relString != "") {
            System.out.println("href=" + hrefString + " rel="external nofollow"  rel="external nofollow"  " + "rel=" + relString);
        }
    }
}

最终结果:

href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow

完整代码:

import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import org.jsoup.Jsoup;




public class Jsoup_Test {

    public static void main(String[] args) throws IOException {
        // TODO 自动生成的方法存根
        
        String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
        Document doc = Jsoup.connect(html).get();
        
        System.out.println(doc);
        Elements elements = doc.getElementsByTag("ul");
        //System.out.println(elements);
        Element tempElement = null;
        for(Element element : elements) {
            if (element.className().equals("nav nav-pills")) {
                tempElement = element;
                //System.out.println(element.className());
                break;
            }
        }
        System.out.println(tempElement);
        Elements li = tempElement.getElementsByTag("li");
        for(Element element : li) {
            Elements element2 = element.getElementsByTag("a");
            for(Element element3 : element2) {
                String hrefString = element3.attr("href");
                String relString = element3.attr("rel");
                if(hrefString != "" && relString != "") {
                    System.out.println("href=" + hrefString + " rel="external nofollow"  rel="external nofollow"  " + "rel=" + relString);
                }
            }
        }        

    }

}

到此这篇关于Java爬虫实现Jsoup利用dom方法遍历Document对象的文章就介绍到这了,更多相关Java Jsoup遍历Document对象内容请搜索编程网以前的文章或继续浏览下面的相关文章希望大家以后多多支持编程网!

阅读原文内容投诉

免责声明:

① 本站未注明“稿件来源”的信息均来自网络整理。其文字、图片和音视频稿件的所属权归原作者所有。本站收集整理出于非商业性的教育和科研之目的,并不意味着本站赞同其观点或证实其内容的真实性。仅作为临时的测试数据,供内部测试之用。本站并未授权任何人以任何方式主动获取本站任何信息。

② 本站未注明“稿件来源”的临时测试数据将在测试完成后最终做删除处理。有问题或投稿请发送至: 邮箱/279061341@qq.com QQ/279061341

软考中级精品资料免费领

  • 历年真题答案解析
  • 备考技巧名师总结
  • 高频考点精准押题
  • 2024年上半年信息系统项目管理师第二批次真题及答案解析(完整版)

    难度     807人已做
    查看
  • 【考后总结】2024年5月26日信息系统项目管理师第2批次考情分析

    难度     351人已做
    查看
  • 【考后总结】2024年5月25日信息系统项目管理师第1批次考情分析

    难度     314人已做
    查看
  • 2024年上半年软考高项第一、二批次真题考点汇总(完整版)

    难度     433人已做
    查看
  • 2024年上半年系统架构设计师考试综合知识真题

    难度     221人已做
    查看

相关文章

发现更多好内容

猜你喜欢

AI推送时光机
位置:首页-资讯-后端开发
咦!没有更多了?去看看其它编程学习网 内容吧
首页课程
资料下载
问答资讯