问Bash命令问题
EN

Stack Overflow用户

提问于 2022-04-11 10:40:12

回答 2查看 81关注 0票数 0

我试图进一步解析我使用一个额外的grep命令生成的输出文件。我目前使用的代码是：

##!/bin/bash

# fetches the links of the movie's imdb pages for a given actor

# fullname="USER INPUT"
read -p "Enter fullname: " fullname

if [ "$fullname" = "Charlie Chaplin" ];
code="nm0000122"
then
code="nm0000050"
fi


curl "https://www.imdb.com/name/$code/#actor" | grep -Eo 
'href="/title/[^"]*' | sed 's#^.*href=\"/#https://www.imdb.com/#g' | 
sort -u > imdb_links.txt

#parses each of the link in the link text file and gets the details for 
each of the movie. THis is followed by the cleaning process
for i in $(cat imdb_links.txt) 
do 
   curl $i | 
   html2text | 
   sed -n '/Sign_In/,$p'|  
   sed -n '/YOUR RATING/q;p' | 
   head -n-1 | 
   tail -n+2 
done > imdb_all.txt

生成的示例输出如下：

EN
⁰
    * Fully supported
    * English (United States)
    * Partially_supported
    * FranÃ§ais (Canada)
    * FranÃ§ais (France)
    * Deutsch (Deutschland)
    * à¤¹à¤¿à¤‚à¤¦à¥€ (à¤à¤¾à¤°à¤¤)
    * Italiano (Italia)
    * PortuguÃªs (Brasil)
    * EspaÃ±ol (EspaÃ±a)
    * EspaÃ±ol (MÃ©xico)
****** Duck Soup ******
    * 19331933
    * Not_RatedNot Rated
    * 1h 9m
IMDb RATING
7.8/10

如何更改代码以进一步解析输出，以便只从电影标题中获取数据直到imdb分级(在本例中，包含标题“Duck Soup”的行一直到结尾)。

web-scraping

sed

bash

人脸核身产品特惠

用户政务民生、金融、通信运营商等场景，接入全面，安全可靠新用户3折起

回答 2

Stack Overflow用户

回答已采纳

发布于 2022-04-11 11:40:56

以下是代码：

#!/bin/bash

# fullname="USER INPUT"
read -p "Enter fullname: " fullname

if [ "$fullname" = "Charlie Chaplin" ]; then
  code="nm0000122"
else
  code="nm0000050"
fi

rm -f imdb_links.txt

curl "https://www.imdb.com/name/$code/#actor" |
  grep -Eo 'href="/title/[^"]*' |
  sed 's#^href="#https://www.imdb.com#g' |
  sort -u |
while read link; do
   # uncomment the next line to save links into file:
   #echo "$link" >>imdb_links.txt

   curl "$link" |
     html2text -utf8 |
     sed -n '/Sign_In/,/YOUR RATING/ p' |
     sed -n '$d; /^\*\{6\}.*\*\{6\}$/,$ p'
done >imdb_all.txt

票数 1

Stack Overflow用户

发布于 2022-04-23 15:58:58

请(！)看看下面的urls，为什么用sed解析HTML是个坏主意

您要做的事情可以使用HTML/XML/JSON解析器西德尔完成，只需一个调用即可完成！

在本例中，我将使用查理卓别林的IMDB作为源代码。

提取所有94个“演员”IMDB电影urls：

$ xidel -s "https://www.imdb.com/name/nm0000122" -e '
  //div[@id="filmo-head-actor"]/following-sibling::div[1]//a/@href
'
/title/tt0061523/?ref_=nm_flmg_act_1
/title/tt0050598/?ref_=nm_flmg_act_2
/title/tt0044837/?ref_=nm_flmg_act_3
[...]
/title/tt0004288/?ref_=nm_flmg_act_94

没有必要将这些保存到文本文件中。只需使用-f (--follow)而不是-e，xidel就会打开它们。

对于单个电影urls，您可以解析HTML以获得所需的文本节点.

$ xidel -s "https://www.imdb.com/title/tt0061523/?ref_=nm_flmg_act_1" -e '
  //h1,
  //div[@class="sc-94726ce4-3 eSKKHi"]/ul/li[1]/span,
  //div[@class="sc-94726ce4-3 eSKKHi"]/ul/li[3],
  (//div[@class="sc-7ab21ed2-2 kYEdvH"])[1]
'
A Countess from Hong Kong
1967
2h
6.0/10

...but和class-names，我想说，这是一个相当脆弱的努力。相反，我建议解析<script>-node中HTML源代码顶部的JSON：

$ xidel -s "https://www.imdb.com/title/tt0061523/?ref_=nm_flmg_act_1" -e '
  parse-json(//script[@type="application/ld+json"])/(
    name,
    datePublished,
    duration,
    aggregateRating/ratingValue
  )
'
A Countess from Hong Kong
1967-03-15
PT2H
6

...or以获得类似的输出，如下所示：

$ xidel -s "https://www.imdb.com/title/tt0061523/?ref_=nm_flmg_act_1" -e '
  parse-json(//script[@type="application/ld+json"])/(
    name,
    year-from-date(date(datePublished)),
    substring(lower-case(duration),3),
    format-number(aggregateRating/ratingValue,"#.0")||"/10"
  )
'
A Countess from Hong Kong
1967
2h
6.0/10

全部合并：

$ xidel -s "https://www.imdb.com/name/nm0000122" \
  -f '//div[@id="filmo-head-actor"]/following-sibling::div[1]//a/@href' \
  -e '
    parse-json(//script[@type="application/ld+json"])/(
      name,
      year-from-date(date(datePublished)),
      substring(lower-case(duration),3),
      format-number(aggregateRating/ratingValue,"#.0")||"/10"
    )
  '
A Countess from Hong Kong
1967
2h
6.0/10
A King in New York
1957
1h50m
7.0/10
Limelight
1952
2h17m
8.0/10
[...]
Making a Living
1914
11m
5.5/10