[리눅스] wget 실습

브라우저에서 다운로드 하는 것 처럼 user-agent 정보 보내기

 

$ wget --user-agent="user-agent 정보" <다운로드할 url>

 

몇몇 웹사이트는 브라우저로 다운로드하지 않는다고 판단될 경우 다운로드를 허가하지 않는 경우가 있다. 이 때 '--user-agent' 옵션으로 user-agent 정보를 함께 보낼 수 있습니다.

 

 

user-agent 참고자료

 

User-agent 정확하게 해석하기 (velog.io)

User-agent 정확하게 해석하기

크롬 user-Agent 값 확인하는 법 | 블로그한경닷컴 (hangyeong.com)

크롬 user-Agent 값 확인하는 법

 

root@DESKTOP-RJ31OF5:/var/cache/apt/archives# wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36" https://blog.hangyeong.com/1009
--2021-12-15 11:34:17--  https://blog.hangyeong.com/1009
Resolving blog.hangyeong.com (blog.hangyeong.com)... 211.249.222.34
Connecting to blog.hangyeong.com (blog.hangyeong.com)|211.249.222.34|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘1009.1’

1009.1                       [ <=>                               ]  58.55K  --.-KB/s    in 0.005s

2021-12-15 11:34:17 (10.6 MB/s) - ‘1009.1’ saved [59956]

root@DESKTOP-RJ31OF5:/var/cache/apt/archives# ls -l | grep 1009.1
-rw-r--r-- 1 root root    59956 Dec 15 11:34 1009.1
root@DESKTOP-RJ31OF5:/var/cache/apt/archives# ls -la | grep 1009.1
-rw-r--r-- 1 root root    59956 Dec 15 11:34 1009.1

 

다운로드 가능한지 확인하기

 

$wget --spider <확인할 url>

 

특정 시간에 다운로드 일정이 잡혀있을 때, 원격지에 파일이 존재하는지 아닌지 확인을 해봐야 합니다. 이 때 '--spider' 옵션을 사용하여 링크가 올바른지 확인합니다.

 

URL이 올바를 경우 : 아랫줄에 Remote file exists 출력됨

에러가 있을 경우 : 아랫줄에 Remote file does not exist -- broken link!!! 출력 됨

 

올바른 경우

root@DESKTOP-RJ31OF5:/var/cache/apt/archives# wget --spider http://kr.archive.ubuntu.com/ubuntu/pool/universe/a/axel/axel_2.17.5-1ubuntu1_amd64.deb
Spider mode enabled. Check if remote file exists.
--2021-12-15 11:51:28--  http://kr.archive.ubuntu.com/ubuntu/pool/universe/a/axel/axel_2.17.5-1ubuntu1_amd64.deb
Resolving kr.archive.ubuntu.com (kr.archive.ubuntu.com)... 91.189.91.38, 91.189.88.142, 91.189.91.39, ...
Connecting to kr.archive.ubuntu.com (kr.archive.ubuntu.com)|91.189.91.38|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 58668 (57K) [application/x-debian-package]
Remote file exists.

root@DESKTOP-RJ31OF5:/var/cache/apt/archives#

에러가 있을 경우

root@DESKTOP-RJ31OF5:/var/cache/apt/archives# wget --spider http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Spider mode enabled. Check if remote file exists.
--2021-12-15 11:44:59--  http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Resolving www.openss7.org (www.openss7.org)... 142.59.210.7
Connecting to www.openss7.org (www.openss7.org)|142.59.210.7|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
Remote file does not exist -- broken link!!!

 

재시도 횟수 지정하기

 

$ wget --tries=75 <시도할 url>

 

 

인터넷 연결에 문제가 있을 경우나 파일 다운로드 실패가 일어날 경우 기본 재시도 횟수는 20 입니다. 하지만 더 오래 시도해볼 필요가 있을 때 '--tries' 옵션으로 재시도 횟수를 조정할 수 있습니다.

 

root@DESKTOP-RJ31OF5:/var/cache/apt/archives# wget --tries=21 http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
--2021-12-15 12:02:54--  http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Resolving www.openss7.org (www.openss7.org)... 142.59.210.7
Connecting to www.openss7.org (www.openss7.org)|142.59.210.7|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-12-15 12:02:54 ERROR 404: Not Found.

 

여러개의 파일 다운로드하기

 

여러 파일을 다운로드 하려면 다운로드 하려는 여러 링크를 써놓은 파일(엔터로 구분)을 만들고 '-i' 옵션을 통해 작업을 진행합니다.

 

1. $ vi downlist.txt

2. vi 에디터에 링크들 입력후 저장

  https://codecoco.tistory.com/55
  https://codecoco.tistory.com/56
  https://codecoco.tistory.com/57

 

3. $ wget -i downlist.txt

 

55,56,57 파일이 다운로드 되었다.

root@DESKTOP-RJ31OF5:/home/ubuntu# vi downlist.txt
root@DESKTOP-RJ31OF5:/home/ubuntu# cat downlist.txt
https://codecoco.tistory.com/55
https://codecoco.tistory.com/56
https://codecoco.tistory.com/57
root@DESKTOP-RJ31OF5:/home/ubuntu# wget -i downlist.txt
--2021-12-15 12:51:23--  https://codecoco.tistory.com/55
Resolving codecoco.tistory.com (codecoco.tistory.com)... 211.231.99.250
Connecting to codecoco.tistory.com (codecoco.tistory.com)|211.231.99.250|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘55’

55                           [ <=>                               ]  48.01K  --.-KB/s    in 0.003s

2021-12-15 12:51:23 (17.0 MB/s) - ‘55’ saved [49166]

--2021-12-15 12:51:23--  https://codecoco.tistory.com/56
Reusing existing connection to codecoco.tistory.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘56’

56                           [ <=>                               ]  54.21K  --.-KB/s    in 0.001s

2021-12-15 12:51:23 (51.4 MB/s) - ‘56’ saved [55506]

--2021-12-15 12:51:23--  https://codecoco.tistory.com/57
Reusing existing connection to codecoco.tistory.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘57’

57                           [ <=>                               ]  51.05K  --.-KB/s    in 0.003s

2021-12-15 12:51:23 (16.9 MB/s) - ‘57’ saved [52280]

FINISHED --2021-12-15 12:51:23--
Total wall clock time: 0.4s
Downloaded: 3 files, 153K in 0.007s (22.2 MB/s)
root@DESKTOP-RJ31OF5:/home/ubuntu# ls -la
total 100340
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 15 12:51  .
drwxr-xr-x 1 root   root        4096 Dec 14 09:55  ..
-rw------- 1 ubuntu ubuntu       916 Dec 15 10:44  .bash_history
-rw-r--r-- 1 ubuntu ubuntu       220 Dec 13 20:13  .bash_logout
-rw-r--r-- 1 ubuntu ubuntu      3968 Dec 14 02:28  .bashrc
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 14 23:02  .cache
drwx------ 1 ubuntu ubuntu      4096 Dec 14 09:46  .config
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 13 20:13  .landscape
-rw-r--r-- 1 ubuntu ubuntu         0 Dec 15 09:31  .motd_shown
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 14 14:10  .npm
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 14 02:29  .nvm
-rw-r--r-- 1 ubuntu ubuntu       807 Dec 13 20:13  .profile
-rw-r--r-- 1 ubuntu ubuntu         0 Dec 13 20:17  .sudo_as_admin_successful
-rw------- 1 ubuntu ubuntu       684 Dec 14 13:10  .viminfo
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 14 02:15  .vscode-server
-rw-r--r-- 1 ubuntu ubuntu       183 Dec 14 02:15  .wget-hsts
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 14 23:02  .yarn
-rw-r--r-- 1 root   root       49166 Dec 15 12:51  55
-rw-r--r-- 1 root   root       55506 Dec 15 12:51  56
-rw-r--r-- 1 root   root       52280 Dec 15 12:51  57
-rw-r--r-- 1 root   root        3275 Dec 15 12:31  README.TXT
drwxr-xr-x 1 root   root        4096 Dec 14 12:37  aaa
-rw-r--r-- 1 root   root          96 Dec 15 12:49  downlist.txt
-rw-r--r-- 1 root   root       50243 Dec 15 10:31 'download_script.php?src_id=7701'
-rw-r--r-- 1 root   root         445 Dec 15 12:25  foo.txt
-rw-r--r-- 1 root   root       16435 Dec 15 10:23  index.html
-rw-r--r-- 1 root   root   102167060 Jun 16  2018  linux-4.17.2.tar.xz
drwxr-xr-x 1 ubuntu ubuntu      4096 Dec 14 23:02  login
-rw-r--r-- 1 root   root       50243 Dec 15 10:32  taglist.zip
-rw-r--r-- 1 root   root         321 Dec 15 10:37  wget-log
-rw-r--r-- 1 root   root         322 Dec 15 10:43  wget-log.1
root@DESKTOP-RJ31OF5:/home/ubuntu#

 

ftp 다운로드하기

 

$ wget <시도할 FTP-URL>

 

wget으로 ftp 다운로드를 할 수 있습니다.

 

root@DESKTOP-RJ31OF5:/home/ubuntu# wget ftp://ftpmirror.your.org/pub/misc/ftp.microsoft.com/developr/TAPI/README.TXT
--2021-12-15 12:30:58--  ftp://ftpmirror.your.org/pub/misc/ftp.microsoft.com/developr/TAPI/README.TXT
           => ‘README.TXT’
Resolving ftpmirror.your.org (ftpmirror.your.org)... 204.9.55.82, 2001:4978:1:420::cc09:3752
Connecting to ftpmirror.your.org (ftpmirror.your.org)|204.9.55.82|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub/misc/ftp.microsoft.com/developr/TAPI ... done.
==> SIZE README.TXT ... 3275
==> PASV ... done.    ==> RETR README.TXT ... done.
Length: 3275 (3.2K) (unauthoritative)

README.TXT               100%[==================================>]   3.20K  --.-KB/s    in 0s

2021-12-15 12:31:01 (210 MB/s) - ‘README.TXT’ saved [3275]