Uploading Files to Internet Remotely via Mac Terminal

Uploading files to Google Drive directly from the Last (using Roll)

An alternative method of pushing data from a calculator to the deject.

In many cases, it may exist hard to ship data from a new machine to another. Examples of this include HPC facilities which are hidden behind a login portal but do not allow ssh tunnelling or simple headless machines which simply accept a few cadre programs installed. In this article, we look at using cURL (a command-line program for transferring data) to push button a zipped file (containing log files) onto our google drive account for further analysis.

Photograph by Element5 Digital on Unsplash

Installation

Most machines volition come with coil installed (endeavor typing which curl). If this is not the case we can install it with

            sudo apt install curl # Linux Debian/Ubuntu          

or

            mash install roll     # Mac          

Now nosotros have it installed, we can wait at creating the credentials needed to send files.

Create your project credentials

Every bit we are allowing access to our google bulldoze, nosotros want to be able to manage this. This is done by creating a project with user-defined permissions to act equally the proxy between our users (in this case us on a different machine) and our account. We first past going to the following page (link below) and creating a new project.

Later on this has been washed we select the Credentials tab (on the left) and "create credentials" from the top.

When asked for the app type, we select Telly and other.

Finally, this generates a client id and a client seacret .

This is your username and password so copy it somewhere secure.

Now we need to verify the device

To do this we ssh into the machine we wish to upload from and run the post-obit command:

            scroll -d "client_id=<client_id>&telescopic=              https://www.googleapis.com/auth/drive.file              " https://oauth2.googleapis.com/device/code          

Doing so we get a response in the following format

            {"device_code": "<long string>",
"user_code": "thirty-30-xxx",
"expires_in": 1800,
"interval": 5,
"verification_url": "https://www.google.com/device"}          

Here we need to visit the URL (https://www.google.com/device) and provide the user code to complete our verification. We at present continue to select our google business relationship and grant the relevant permissions. When doing this brand sure to note down the device code for the next stride.

Get Bearer code

When we start uploading, this is the code we shall need to use to identify our business relationship. We get it past using the post-obit:

            curl -d client_id=<client id>              -d client_secret=<client cloak-and-dagger>              -d device_code=<device code>              -d grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code https://accounts.google.com/o/oauth2/token          

The client id and hugger-mugger are saved from the first footstep, and the device code in the previous section. The output should be in the format:

            {
"access_token": ".....",
"expires_in": 3599,
"refresh_token": "....",
"scope": "https://www.googleapis.com/auth/bulldoze.file",
"token_type": "Bearer"
}          

Write down the access_token as information technology will be needed in the upload stage.

Upload files

The command we utilise to upload files is given below

            curl -X POST -L \
              -H "Dominance: Bearer              <enter                            access token here              >" \
              -F "metadata={name :'<our.zip>'};type=application/json;charset=UTF-8" \
              -F "file=@<our.null>;type=awarding/null" \
              "https://www.googleapis.com/upload/bulldoze/v3/files?uploadType=multipart"          

Here you may need to enable the app API before being allowed to upload information. The link to do this is given in the error message if this is the case.

Here multipart files are expected to only be a couple of MB in size. However if you are looking at moving larger files resumable may be better suited (see https://developers.google.com/drive/api/v3/manage-uploads )

Wrap it all up in a script

Now nosotros know our commands piece of work nosotros can create an executable script to practise all the work for us. Hither we can provide a group of files, it zips them upward and then sends them to google drive.

Nosotros first past creating a new file with nano curlgoogle; and enter the following code — remember to add together your own personal auth token! Python 2.7 has been chosen equally this is notwithstanding the default python version on older systems, however the script below should also run for python 3.

It should require no new dependencies provided roll already exists on the system.

            #!/usr/bin/python            '''
A quick python script to automate whorl->googledrive interfacing
This should require zilch more than than the organization python version and curl. Written for python2.7 (with three in mind).            Dan Ellis 2020
'''            import os,sys,json            if sys.version[0]=='iii':
              raw_input = lambda(ten): input(ten)            ##############################
#Owner data goes here!#
##############################
name = 'curldata'
client_id= '<enter your client id>'
client_secret='<enter your customer secret>'            ##############################                          
cmd1 = json.loads(os.popen('whorl -d "client_id=%s&scope=https://www.googleapis.com/auth/bulldoze.file" https://oauth2.googleapis.com/device/code'%client_id).read())            str(raw_input('\northward Enter %(user_code)southward\n\north at %(verification_url)due south \n\n Then striking Enter to continue.'%cmd1))            str(raw_input('(twice)'))            cmd2 = json.loads(os.popen(('curl -d client_id=%s -d client_secret=%s -d device_code=%s -d grant_type=urn~~3Aietf~~3Aparams~~3Aoauth~~3Agrant-type~~3Adevice_code https://accounts.google.com/o/oauth2/token'%(client_id,client_secret,cmd1['device_code'])).replace('~~','%')).read())
impress(cmd2)            # nada files
cmd3 = bone.popen('zip -r %southward.zippo %s'%(name,' '.join(sys.argv[i:]))).read
print(cmd3)            cmd4 = os.popen('''
curlicue -X POST -50 \
              -H "Authorization: Bearer %due south" \
              -F "metadata={name :\'%s\'};type=awarding/json;charset=UTF-8" \
              -F "file=@%s.cypher;type=application/zip" \
              "https://world wide web.googleapis.com/upload/drive/v3/files?uploadType=multipart"
              '''%(cmd2["access_token"],name,name)).read()            print(cmd4)
print('end')          

We then brand it executable chmod a+x curlgoogle allowing us to use information technology in an executable manner:

                          ./curlgoogle              file1 file2.txt file3.jpg etc...                      

Conclusions

And at that place we accept it, an easy way to send multiple log files from a headless machine to a google bulldoze repository, which can exist accessed by multiple people for analysis.

If you are in demand of more than information, stack overflow answers by Tanaike and HAKS (amongst others) were particularly helpful in creating this post.

worthyknines.blogspot.com

Source: https://towardsdatascience.com/uploading-files-to-google-drive-directly-from-the-terminal-using-curl-2b89db28bb06

Share this post