Download GB’s of Data in Seconds
While working with real time data,loading and unloading of data on your system is most curious task.
real time data is in TBs, and if we start downloading it by normal way as we doing always it will take couple of hours ,sometimes days which any industry cant afford to do in todays world.
So,is there any way to do this in fast??
and answer is yes!!!
just follow the steps in order to download the data in few seconds only…
- Open your Chrome browser and install CurlWget extension on it
follow the link to download the extension: https://chrome.google.com/webstore/detail/curlwget/jmocjfidanebdlinpbcdkcmgdifblncg?hl=en
2.Now go to the data downloading link and start downloading the data:
Wait..Wait…Wait…
Here , I am downloading the 17 GB of data. and if i continue same it will take almost 5–6 hours to download it and what about the uploading of data, it again take around 5–6 hrs of time to upload on google colab or jupyter notebook
3. So start downloading the data and hit “cancel” button
4.Then go to the curlWget extension we downloaded and click on it,you will see following command:
copy all the data in green window:
5.Open google colab and new cell in it:
first enter “!” mark and then paste the data you copied from the curlWget extension
and hit enter :
— 2020–01–13 08:16:14 — https://storage.googleapis.com/kagglesdsdata/competitions/4117/46665/train.7z?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=1579160772&Signature=KAOXcH43vapAEGt0T%2BqqrIg9EWv1zn4jnE6MM0RKQfLLPF%2FcRYOuPHewOsXtNZ4RIjHHzJZsyS09boK%2Fee6ICkmS2N4Tq1nRbQbb93MPNqt21jbYtY15ryHpPqQ9nti2%2Fa03Y%2Fcni6dieeQ%2Fg2m8p9jxjuPj3hJ7vSHZwddNH6cJn%2B64vDGlzMzFFjxj%2Faz2IdbSJdmZTWvFYTvLeLB%2FgiWqiCJZjs8fI20wohpzgR8OT6PU5IM1k%2Bm25wCSyuY%2B4WysyX3jOGaFWnWasZTUPO1Dbcwir2BNhyKFBNaEJasUa3WE7pvawNXxLaRTMGYQXa8U1AhuJLu%2Bd5TAZAgY3A%3D%3D&response-content-disposition=attachment%3B+filename%3Dtrain.7z Resolving storage.googleapis.com (storage.googleapis.com)… 74.125.197.128, 2607:f8b0:400e:c07::80 Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.197.128|:443… connected. HTTP request sent, awaiting response… 200 OK Length: 18810691091 (18G) [application/x-7z-compressed] Saving to: ‘train.7z’ train.7z 100%[===================>] 17.52G 77.7MB/s in 4m 58s 2020–01–13 08:21:17 (60.3 MB/s) — ‘train.7z’ saved [18810691091/18810691091]you will see data has been downloded in just 2 min:
isn’t it amazing!!!!….
Blog by:
Akshay Bhor..
an Data Scientist..
