Anyone involved with web development, web application programming, creating REST APIs, or interfacing with REST APIs is going to have the curl command in their tool belt. It is an extremely versatile tool that can talk to many different protocols.
This post will be an ever growing list of curl commands that I continually use throughout my projects.
Get HTTP Headers Only
Grab only the HTTP headers, instead of the entire HTTP response and content, from a URL:
curl -I $URL
To ensure you see the entire request/response chain, add the -L command line switch to follow redirects:
curl -IL $URL
Check HTTP Redirects
If you maintain a lot of domain redirects, it is important to make sure they continually work. The following curl command can be used to loop through a text file containing URLs to check. The output for each check will be the final HTTP status code and URL.
curl -sL -w "%{http_code} %{url_effective}\\n" "$URL" -o /dev/null
For example, to verify http://thornelaboratories.net redirects to https://thornelabs.net:
curl -sL -w "%{http_code} %{url_effective}\\n" "http://thornelaboratories.net" -o /dev/null
Will output:
200 https://thornelabs.net/
Pass Custom Host Header
If you do local web development, you probably have to frequently access your website or web application from localhost. In most situations this shouldn’t be a problem, but for those situations where it is (such as testing Apache VirtualHosts or nginx Server Blocks), you can easily change the Host header with curl to tell the website or web application where you want to go.
curl -i -H 'Host: example.com' $URL
Alternatively, temporarily add a record in your workstation’s /etc/hosts file pointing the domain name of the URL you are accessing to 127.0.0.1.
Force Request through a Specific IP Address
Instead of changing your /etc/hosts file, force a curl request to go through an IP address that differs from what the hostname resolves to in public DNS.
curl -IL $HOSTNAME --resolve $HOSTNAME:$PORT:$SPECIFIC-IP
POST Data and Content Type to URL
POST plain text data to URL:
curl -X POST -d 'plain-text' -H "Content-Type:text/plain" $URL
POST JSON data to URL:
curl -X POST -d '{"key1":"value1","key2":"value2"}' -H "Content-Type:application/json" $URL
Pass JSON Payload to curl Using a File, Bash Variable, or HERE Document
File
Create a file called payload, put valid JSON in it, and reference it with the curl command using the following command:
curl -d @./payload -H "X-Auth: $TOKEN" "https://api.example.com/api/query"
Bash Variable
If you are using the curl command within a shell script and want to pass JSON to it from a Bash variable in the same shell script, use the following method:
PAYLOAD='
[{
"auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "USERNAME",
"domain": {
"id": "default"
},
"password": "PASSWORD"
}
}
},
"scope": {
"project": {
"name": "PROJECT",
"domain": {
"id": "default"
}
}
}
}
}]'
RESULT=$(curl -s -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<< "$PAYLOAD")
HERE Document
Another method you can use is a HERE document. This eliminates the need for the $PAYLOAD Bash variable.
RESULT=$(curl -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<PAYLOAD
[{
"auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "USERNAME",
"domain": {
"id": "default"
},
"password": "PASSWORD"
}
}
},
"scope": {
"project": {
"name": "PROJECT",
"domain": {
"id": "default"
}
}
}
}
}]
PAYLOAD)
Using curl to Query an API
Many of the tools used today are built on top of some sort of API. Those tools abstract all of the granular details and information that comes through an API request and response. Sometimes it is useful to see what goes on behind the scenes, especially when it comes time to troubleshoot.
The following example uses the curl command to query the OpenStack API to show the details about a particular hypervisor managed by OpenStack Nova.
Get a token
Before you can do anything with any API, you need a token. With OpenStack it’s no different. OpenStack Keystone is the identity and authentication service that is used to generate tokens for end users and services.
The Keystone v2 API is quickly being deprecated. I have only included it for posterity’s sake, so jump to the v3 API below to generate a token:
curl -s \
-X POST https://10.240.0.100:5000/v2.0/tokens \
-H "Content-Type: application/json" \
-d '{"auth": {"tenantName": "TENANT", "passwordCredentials":{"username": "USERNAME", "password": "PASSWORD"}}}'
What follows is the Keystone v3 API. You will need a valid username and password already stored in OpenStack Keystone to generate a token:
curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "USERNAME",
"domain": { "id": "default" },
"password": "PASSWORD"
}
}
},
"scope": {
"project": {
"name": "PROJECT",
"domain": { "id": "default" }
}
}
}
}' \
https://10.240.0.100:5000/v3/auth/tokens
This will return a lot of JSON. Unfortunately, I did not make a copy of that JSON output when I originally created these notes. However, within that JSON will be a token that will be used in all of the subsequent commands. Let’s assume the token returned is 1234567890abcdefghijklmnopqrstuv.
Now that a token has been generated, I can begin querying the OpenStack Nova API for details on the particular hypervisor. For this example, I want details about compute01.local.lan, and I’m going to need its id:
curl -H "X-Auth-Token:1234567890abcdefghijklmnopqrstuv" http://10.240.0.100:8774/v3/os-hypervisors/compute01.local.lan/search | python -m json.tool
Take note, the curl command is going to output JSON which can be difficult to read. I am piping the output to python -m json.tool to make it easier to read.
The above command returns the following JSON:
{
"hypervisors": [
{
"hypervisor_hostname": "compute01.local.lan",
"id": 1,
"state": "up",
"status": "enabled"
}
]
}
Now that I have the id, I can query the details about compute01.local.lan:
curl -H "X-Auth-Token:1234567890abcdefghijklmnopqrstuv" http://10.240.0.100:8774/v3/os-hypervisors/1 | python -m json.tool
The above command returns the following JSON which provides all of the details for that particular hypervisor:
{
"hypervisor": {
"cpu_info": "{\"vendor\": \"Intel\", \"model\": \"SandyBridge\", \"arch\": \"x86_64\", \"features\": [\"ssse3\", \"pge\", \"avx\", \"clflush\", \"sep\", \"syscall\", \"vme\", \"dtes64\", \"tsc\", \"xsave\", \"vmx\", \"xtpr\", \"cmov\", \"pcid\", \"est\", \"pat\", \"monitor\", \"smx\", \"lm\", \"msr\", \"nx\", \"fxsr\", \"tm\", \"sse4.1\", \"pae\", \"sse4.2\", \"pclmuldq\", \"acpi\", \"tsc-deadline\", \"mmx\", \"osxsave\", \"cx8\", \"mce\", \"mtrr\", \"rdtscp\", \"ht\", \"dca\", \"lahf_lm\", \"pdcm\", \"mca\", \"pdpe1gb\", \"apic\", \"sse\", \"pse\", \"ds\", \"pni\", \"tm2\", \"aes\", \"sse2\", \"ss\", \"pbe\", \"de\", \"fpu\", \"cx16\", \"pse36\", \"ds_cpl\", \"popcnt\", \"x2apic\"], \"topology\": {\"cores\": 6, \"threads\": 2, \"sockets\": 2}}",
"current_workload": 0,
"disk_available_least": 752,
"free_disk_gb": 878,
"free_ram_mb": 117633,
"host_ip": "10.240.0.200",
"hypervisor_hostname": "compute01.local.lan",
"hypervisor_type": "QEMU",
"hypervisor_version": 2000000,
"id": 1,
"local_gb": 971,
"local_gb_used": 93,
"memory_mb": 128897,
"memory_mb_used": 11264,
"os-pci:pci_stats": [],
"running_vms": 5,
"service": {
"disabled_reason": null,
"host": "compute01",
"id": 25
},
"state": "up",
"status": "enabled",
"vcpus": 24,
"vcpus_used": 6
}
}
Google Cloud curl, or gcurl
To interact with Google Cloud APIs using curl, you have to pass a token in the Authorization request header. This is tedious to do manually every time and the token will eventually expire, so you can create a command line alias to make this process easier. The generated token will map to the Google account you currently have authenticated with gcloud (verify which Google account is authenticated with the gcloud config list command).
The following command is copied from the Google Cloud Run Authenticating developers, services, and end users documentation:
alias gcurl='curl --header "Authorization: Bearer $(gcloud config config-helper --format=value\(credential.access_token\) --force-auth-refresh)"'
