curl Commands Cheat Sheet

Tuesday, March 7, 2017

Anyone involved with web development, web application programming, creating REST APIs, or interfacing with REST APIs is going to have the curl command in their tool belt. It is an extremely versatile tool that can talk to many different protocols.

This post will be an ever growing list of curl commands that I continually use throughout my projects.

Get HTTP Headers Only

Grab only the HTTP headers, instead of the entire HTTP response and content, from a URL:

curl -I $URL

Check HTTP Redirects

If you maintain a lot of domain redirects, it is important to make sure they continually work. The following curl command can be used to loop through a text file containing URLs to check. The output for each check will be the final HTTP status code and URL.

curl -sL -w "%{http_code} %{url_effective}\\n" "$URL" -o /dev/null

For example, to verify http://thornelaboratories.net redirects to https://thornelabs.net:

curl -sL -w "%{http_code} %{url_effective}\\n" "http://thornelaboratories.net" -o /dev/null

Will output:

200 https://thornelabs.net/

Pass Custom Host Header

If you do local web development, you probably have to frequently access your website or web application from localhost. In most situations this shouldn’t be a problem, but for those situations where it is (such as testing Apache VirtualHosts or nginx Server Blocks), you can easily change the Host header with curl to tell the website or web application where you want to go.

curl -i -H 'Host: example.com' $URL

Alternatively, temporarily add a record in your workstation’s /etc/hosts file pointing the domain name of the URL you are accessing to 127.0.0.1.

POST Data and Content Type to URL

POST plain text data to URL:

curl -X POST -d 'plain-text' -H "Content-Type:text/plain" $URL

POST JSON data to URL:

curl -X POST -d '{"key1":"value1","key2":"value2"}' -H "Content-Type:application/json" $URL

Pass JSON Payload to curl Using a File, Bash Variable, or HERE Document

File

Create a file called payload, put valid JSON in it, and reference it with the curl command using the following command:

curl -d @./payload -H "X-Auth: $TOKEN" "https://api.example.com/api/query"

Bash Variable

If you are using the curl command within a shell script and want to pass JSON to it from a Bash variable in the same shell script, use the following method:

PAYLOAD='
[{
    "auth": {
        "identity": {
            "methods": ["password"],
            "password": {
                "user": {
                    "name": "USERNAME",
                    "domain": {
                        "id": "default"
                    },
                    "password": "PASSWORD"
                }
            }
        },
        "scope": {
            "project": {
                "name": "PROJECT",
                "domain": {
                    "id": "default"
                }
            }
        }
    }
}]'

RESULT=$(curl -s -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<< "$PAYLOAD")

HERE Document

Another method you can use is a HERE document. This eliminates the need for the $PAYLOAD Bash variable.

RESULT=$(curl -d @- -H "X-Auth: $TOKEN" "https://api.example.com/api/query" <<PAYLOAD
[{
    "auth": {
        "identity": {
            "methods": ["password"],
            "password": {
                "user": {
                    "name": "USERNAME",
                    "domain": {
                        "id": "default"
                    },
                    "password": "PASSWORD"
                }
            }
        },
        "scope": {
            "project": {
                "name": "PROJECT",
                "domain": {
                    "id": "default"
                }
            }
        }
    }
}]
PAYLOAD)

Using curl to Query an API

Many of the tools used today are built on top of some sort of API. Those tools abstract all of the granular details and information that comes through an API request and response. Sometimes it is useful to see what goes on behind the scenes, especially when it comes time to troubleshoot.

The following example uses the curl command to query the OpenStack API to show the details about a particular hypervisor managed by OpenStack Nova.

Get a token

Before you can do anything with any API, you need a token. With OpenStack it’s no different. OpenStack Keystone is the identity and authentication service that is used to generate tokens for end users and services.

The Keystone v2 API is quickly being deprecated. I have only included it for posterity’s sake, so jump to the v3 API below to generate a token.

curl -s \
     -X POST https://10.240.0.100:5000/v2.0/tokens \
     -H "Content-Type: application/json" \
     -d '{"auth": {"tenantName": "TENANT", "passwordCredentials":{"username": "USERNAME", "password": "PASSWORD"}}}'

What follows is the Keystone v3 API. You will need a valid username and password already stored in OpenStack Keystone to generate a token:

curl -i \
     -H "Content-Type: application/json" \
     -d '
        { "auth": {
            "identity": {
              "methods": ["password"],
              "password": {
                "user": {
                  "name": "USERNAME",
                  "domain": { "id": "default" },
                  "password": "PASSWORD"
                }
              }
            },
            "scope": {
              "project": {
                "name": "PROJECT",
                "domain": { "id": "default" }
              }
            }
          }
        }' \
     https://10.240.0.100:5000/v3/auth/tokens

This will return a lot of JSON. Unfortunately, I did not make a copy of that JSON output when I originally created these notes. However, within that JSON will be a token that will be used in all of the subsequent commands. Let’s assume the token returned is 1234567890abcdefghijklmnopqrstuv.

Now that a token has been generated, I can begin querying the OpenStack Nova API for details on the particular hypervisor. For this example, I want details about compute01.local.lan, and I’m going to need its id.

curl -H "X-Auth-Token:1234567890abcdefghijklmnopqrstuv" http://10.240.0.100:8774/v3/os-hypervisors/compute01.local.lan/search | python -m json.tool

Take note, the curl command is going to output JSON which can be difficult to read. I am piping the output to python -m json.tool to make it easier to read.

The above command returns the following JSON:

{
    "hypervisors": [
        {
            "hypervisor_hostname": "compute01.local.lan",
            "id": 1,
            "state": "up",
            "status": "enabled"
        }
    ]
}

Now that I have the id, I can query the details about compute01.local.lan:

curl -H "X-Auth-Token:ffdd416e59c242f699ccb9d55e0e3d09" http://10.240.0.100:8774/v3/os-hypervisors/1 | python -m json.tool

The above command returns the following JSON which provides all of the details for that particular hypervisor.

{
    "hypervisor": {
        "cpu_info": "{\"vendor\": \"Intel\", \"model\": \"SandyBridge\", \"arch\": \"x86_64\", \"features\": [\"ssse3\", \"pge\", \"avx\", \"clflush\", \"sep\", \"syscall\", \"vme\", \"dtes64\", \"tsc\", \"xsave\", \"vmx\", \"xtpr\", \"cmov\", \"pcid\", \"est\", \"pat\", \"monitor\", \"smx\", \"lm\", \"msr\", \"nx\", \"fxsr\", \"tm\", \"sse4.1\", \"pae\", \"sse4.2\", \"pclmuldq\", \"acpi\", \"tsc-deadline\", \"mmx\", \"osxsave\", \"cx8\", \"mce\", \"mtrr\", \"rdtscp\", \"ht\", \"dca\", \"lahf_lm\", \"pdcm\", \"mca\", \"pdpe1gb\", \"apic\", \"sse\", \"pse\", \"ds\", \"pni\", \"tm2\", \"aes\", \"sse2\", \"ss\", \"pbe\", \"de\", \"fpu\", \"cx16\", \"pse36\", \"ds_cpl\", \"popcnt\", \"x2apic\"], \"topology\": {\"cores\": 6, \"threads\": 2, \"sockets\": 2}}",
        "current_workload": 0,
        "disk_available_least": 752,
        "free_disk_gb": 878,
        "free_ram_mb": 117633,
        "host_ip": "10.240.0.200",
        "hypervisor_hostname": "compute01.local.lan",
        "hypervisor_type": "QEMU",
        "hypervisor_version": 2000000,
        "id": 1,
        "local_gb": 971,
        "local_gb_used": 93,
        "memory_mb": 128897,
        "memory_mb_used": 11264,
        "os-pci:pci_stats": [],
        "running_vms": 5,
        "service": {
            "disabled_reason": null,
            "host": "compute01",
            "id": 25
        },
        "state": "up",
        "status": "enabled",
        "vcpus": 24,
        "vcpus_used": 6
    }
}

References

How to pass the value of a variable to the stdin of a command?

comments powered by Disqus