Tool to download large HTML file

Sometimes when you need to download catalog data using a web browser (Chrome, Firefox, etc.), it will take some time and might cause crash or lag. Here is some ways to do it:

I. Using wget
+ Install if you do not have in terminal:

yum install wget

or

sudo apt-get install wget

+ Basic commands:

wget –output-document= {output file}  {link to download}

E.g:wget ‐‐output-document=filename.html example.com

Details of how to use it can be find here or here.

II. A simple python code

All you need to do it install python and tqdm package using

pip install tqdm

Or run the sh file which I have already written the code to install tqdm package.

Run ./download.sh or python download.py to run, example below:

screenshot-from-2016-10-18-14-37-56You can download this small utility here.

Nguyễn Công Nghĩa (IES, Academia Sinica)

2 thoughts on “Tool to download large HTML file

Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Powered by WordPress.com.

Up ↑

%d bloggers like this: