| Online: http://curl.haxx.se/docs/httpscripting.shtml |
| Author: Daniel Stenberg <daniel@haxx.se> |
| Date: August 20, 2001 |
| Version: 0.4 |
| |
| The Art Of Scripting HTTP Requests Using Curl |
| ============================================= |
| |
| This document will assume that you're familiar with HTML and general |
| networking. |
| |
| The possibility to write scripts is essential to make a good computer |
| system. Unix' capability to be extended by shell scripts and various tools to |
| run various automated commands and scripts is one reason why it has succeeded |
| so well. |
| |
| The increasing amount of applications moving to the web has made "HTTP |
| Scripting" more frequently requested and wanted. To be able to automatically |
| extract information from the web, to fake users, to post or upload data to |
| web servers are all important tasks today. |
| |
| Curl is a command line tool for doing all sorts of URL manipulations and |
| transfers, but this particular document will focus on how to use it when |
| doing HTTP requests for fun and profit. I'll assume that you know how to |
| invoke 'curl --help' or 'curl --manual' to get basic information about it. |
| |
| Curl is not written to do everything for you. It makes the requests, it gets |
| the data, it sends data and it retrieves the information. You probably need |
| to glue everything together using some kind of script language or repeated |
| manual invokes. |
| |
| 1. The HTTP Protocol |
| |
| HTTP is the protocol used to fetch data from web servers. It is a very simple |
| protocol that is built upon TCP/IP. The protocol also allows information to |
| get sent to the server from the client using a few different methods, as will |
| be shown here. |
| |
| HTTP is plain ASCII text lines being sent by the client to a server to |
| request a particular action, and then the server replies a few text lines |
| before the actual requested content is sent to the client. |
| |
| Using curl's option -v will display what kind of commands curl sends to the |
| server, as well as a few other informational texts. -v is the single most |
| useful option when it comes to debug or even understand the curl<->server |
| interaction. |
| |
| 2. URL |
| |
| The Uniform Resource Locator format is how you specify the address of a |
| particular resource on the internet. You know these, you've seen URLs like |
| http://curl.haxx.se or https://yourbank.com a million times. |
| |
| 3. GET a page |
| |
| The simplest and most common request/operation made using HTTP is to get a |
| URL. The URL could itself refer to a web page, an image or a file. The client |
| issues a GET request to the server and receives the document it asked for. |
| If you isse the command line |
| |
| curl http://curl.haxx.se |
| |
| you get a web page returned in your terminal window. The entire HTML document |
| that that URL holds. |
| |
| All HTTP replies contain a set of headers that are normally hidden, use |
| curl's -i option to display them as well as the rest of the document. You can |
| also ask the remote server for ONLY the headers by using the -I option. |
| |
| 4. Forms |
| |
| Forms are the general way a web site can present a HTML page with fields for |
| the user to enter data in, and then press some kind of 'OK' or 'submit' |
| button to get that data sent to the server. The server then typically uses |
| the posted data to decide how to act. Like using the entered words to search |
| in a database, or to add the info in a bug track system, display the entered |
| address on a map or using the info as a login-prompt verifying that the user |
| is allowed to see what it is about to see. |
| |
| Of course there has to be some kind of program in the server end to receive |
| the data you send. You cannot just invent something out of the air. |
| |
| 4.1 GET |
| |
| A GET-form uses the method GET, as specified in HTML like: |
| |
| <form method="GET" action="junk.cgi"> |
| <input type=text name="birthyear"> |
| <input type=submit name=press value="OK"> |
| </form> |
| |
| In your favourite browser, this form will appear with a text box to fill in |
| and a press-button labeled "OK". If you fill in '1905' and press the OK |
| button, your browser will then create a new URL to get for you. The URL will |
| get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the |
| previous URL. |
| |
| If the original form was seen on the page "www.hotmail.com/when/birth.html", |
| the second page you'll get will become |
| "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK". |
| |
| Most search engines work this way. |
| |
| To make curl do the GET form post for you, just enter the expected created |
| URL: |
| |
| curl "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK" |
| |
| 4.2 POST |
| |
| The GET method makes all input field names get displayed in the URL field of |
| your browser. That's generally a good thing when you want to be able to |
| bookmark that page with your given data, but it is an obvious disadvantage |
| if you entered secret information in one of the fields or if there are a |
| large amount of fields creating a very long and unreadable URL. |
| |
| The HTTP protocol then offers the POST method. This way the client sends the |
| data separated from the URL and thus you won't see any of it in the URL |
| address field. |
| |
| The form would look very similar to the previous one: |
| |
| <form method="POST" action="junk.cgi"> |
| <input type=text name="birthyear"> |
| <input type=submit name=press value="OK"> |
| </form> |
| |
| And to use curl to post this form with the same data filled in as before, we |
| could do it like: |
| |
| curl -d "birthyear=1905&press=OK" www.hotmail.com/when/junk.cgi |
| |
| This kind of POST will use the Content-Type |
| application/x-www-form-urlencoded and is the most widely used POST kind. |
| |
| 4.3 FILE UPLOAD POST |
| |
| Back in late 1995 they defined a new way to post data over HTTP. It was |
| documented in the RFC 1867, why this method sometimes is refered to as |
| a rfc1867-posting. |
| |
| This method is mainly designed to better support file uploads. A form that |
| allows a user to upload a file could be written like this in HTML: |
| |
| <form method="POST" enctype='multipart/form-data' action="upload.cgi"> |
| <input type=file name=upload> |
| <input type=submit name=press value="OK"> |
| </form> |
| |
| This clearly shows that the Content-Type about to be sent is |
| multipart/form-data. |
| |
| To post to a form like this with curl, you enter a command line like: |
| |
| curl -F upload=@localfilename -F press=OK [URL] |
| |
| 4.4 HIDDEN FIELDS |
| |
| A very common way for HTML based application to pass state information |
| between pages is to add hidden fields to the forms. Hidden fields are |
| already filled in, they aren't displayed to the user and they get passed |
| along just as all the other fields. |
| |
| A similar example form with one visible field, one hidden field and one |
| submit button could look like: |
| |
| <form method="POST" action="foobar.cgi"> |
| <input type=text name="birthyear"> |
| <input type=hidden name="person" value="daniel"> |
| <input type=submit name="press" value="OK"> |
| </form> |
| |
| To post this with curl, you won't have to think about if the fields are |
| hidden or not. To curl they're all the same: |
| |
| curl -d "birthyear=1905&press=OK&person=daniel" [URL] |
| |
| 4.5 FIGURE OUT WHAT A POST LOOKS LIKE |
| |
| When you're about fill in a form and send to a server by using curl instead |
| of a browser, you're of course very interested in sending a POST exactly the |
| way your browser does. |
| |
| An easy way to get to see this, is to save the HTML page with the form on |
| your local disk, mofidy the 'method' to a GET, and press the submit button |
| (you could also change the action URL if you want to). |
| |
| You will then clearly see the data get appended to the URL, separated with a |
| '?'-letter as GET forms are supposed to. |
| |
| 5. PUT |
| |
| The perhaps best way to upload data to a HTTP server is to use PUT. Then |
| again, this of course requires that someone put a program or script on the |
| server end that knows how to receive a HTTP PUT stream. |
| |
| Put a file to a HTTP server with curl: |
| |
| curl -T uploadfile www.uploadhttp.com/receive.cgi |
| |
| 6. AUTHENTICATION |
| |
| Authentication is the ability to tell the server your username and password |
| so that it can verify that you're allowed to do the request you're doing. The |
| basic authentication used in HTTP is *plain* *text* based, which means it |
| sends username and password only slightly obfuscated, but still fully |
| readable by anyone that sniffs on the network between you and the remote |
| server. |
| |
| To tell curl to use a user and password for authentication: |
| |
| curl -u name:password www.secrets.com |
| |
| Sometimes your HTTP access is only available through the use of a HTTP |
| proxy. This seems to be especially common at various companies. A HTTP proxy |
| may require its own user and password to allow the client to get through to |
| the internet. To specify those with curl, run something like: |
| |
| curl -U proxyuser:proxypassword curl.haxx.se |
| |
| If you use any one these user+password options but leave out the password |
| part, curl will prompt for the password interactively. |
| |
| Do note that when a program is run, its parameters are possible to see when |
| listing the running processes of the system. Thus, other users may be able to |
| watch your passwords if you pass them as plain command line options. There |
| are ways to circumvent this. |
| |
| 7. REFERER |
| |
| A HTTP request may include a 'referer' field, which can be used to tell from |
| which URL the client got to this particular resource. Some programs/scripts |
| check the referer field of requests to verify that this wasn't arriving from |
| an external site or an unknown page. While this is a stupid way to check |
| something so easily forged, many scripts still do it. Using curl, you can put |
| anything you want in the referer-field and thus more easily be able to fool |
| the server into serving your request. |
| |
| Use curl to set the referer field with: |
| |
| curl -e http://curl.haxx.se daniel.haxx.se |
| |
| 8. USER AGENT |
| |
| Very similar to the referer field, all HTTP requests may set the User-Agent |
| field. It names what user agent (client) that is being used. Many |
| applications use this information to decide how to display pages. Silly web |
| programmers try to make different pages for users of different browsers to |
| make them look the best possible for their particular browsers. They usually |
| also do different kinds of javascript, vbscript etc. |
| |
| At times, you will see that getting a page with curl will not return the same |
| page that you see when getting the page with your browser. Then you know it |
| is time to set the User Agent field to fool the server into thinking you're |
| one of those browsers. |
| |
| To make curl look like Internet Explorer on a Windows 2000 box: |
| |
| curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL] |
| |
| Or why not look like you're using Netscape 4.73 on a Linux (PIII) box: |
| |
| curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL] |
| |
| 9. REDIRECTS |
| |
| When a resource is requested from a server, the reply from the server may |
| include a hint about where the browser should go next to find this page, or a |
| new page keeping newly generated output. The header that tells the browser |
| to redirect is Location:. |
| |
| Curl does not follow Location: headers by default, but will simply display |
| such pages in the same manner it display all HTTP replies. It does however |
| feature an option that will make it attempt to follow the Location: pointers. |
| |
| To tell curl to follow a Location: |
| |
| curl -L www.sitethatredirects.com |
| |
| If you use curl to POST to a site that immediately redirects you to another |
| page, you can safely use -L and -d/-F together. Curl will only use POST in |
| the first request, and then revert to GET in the following operations. |
| |
| 10. COOKIES |
| |
| The way the web browsers do "client side state control" is by using |
| cookies. Cookies are just names with associated contents. The cookies are |
| sent to the client by the server. The server tells the client for what path |
| and host name it wants the cookie sent back, and it also sends an expiration |
| date and a few more properties. |
| |
| When a client communicates with a server with a name and path as previously |
| specified in a received cookie, the client sends back the cookies and their |
| contents to the server, unless of course they are expired. |
| |
| Many applications and servers use this method to connect a series of requests |
| into a single logical session. To be able to use curl in such occations, we |
| must be able to record and send back cookies the way the web application |
| expects them. The same way browsers deal with them. |
| |
| The simplest way to send a few cookies to the server when getting a page with |
| curl is to add them on the command line like: |
| |
| curl -b "name=Daniel" www.cookiesite.com |
| |
| Cookies are sent as common HTTP headers. This is practical as it allows curl |
| to record cookies simply by recording headers. Record cookies with curl by |
| using the -D option like: |
| |
| curl -D headers_and_cookies www.cookiesite.com |
| |
| Curl has a full blown cookie parsing engine built-in that comes to use if you |
| want to reconnect to a server and use cookies that were stored from a |
| previous connection (or handicrafted manually to fool the server into |
| believing you had a previous connection). To use previously stored cookies, |
| you run curl like: |
| |
| curl -b stored_cookies_in_file www.cookiesite.com |
| |
| Curl's "cookie engine" gets enabled when you use the -b option. If you only |
| want curl to understand received cookies, use -b with a file that doesn't |
| exist. Example, if you want to let curl understand cookies from a page and |
| follow a location (and thus possibly send back cookies it received), you can |
| invoke it like: |
| |
| curl -b nada -L www.cookiesite.com |
| |
| 11. HTTPS |
| |
| There are a few ways to do secure HTTP transfers. The by far most common |
| protocol for doing this is what is generally known as HTTPS, HTTP over |
| SSL. SSL encrypts all the data that is sent and received over the network and |
| thus makes it harder for attackers to spy on sensitive information. |
| |
| SSL (or TLS as the latest version of the standard is called) offers a |
| truckload of advanced features to allow all those encryptions and key |
| infrastructure mechanisms encrypted HTTP requires. |
| |
| Curl supports encrypted fetches thanks to the freely available OpenSSL |
| libraries. To get a page from a HTTPS server, simply run curl like: |
| |
| curl https://that.secure.server.com |
| |
| 11.1 CERTIFICATES |
| |
| In the HTTPS world, you use certificates to validate that you are the one |
| you you claim to be, as an addition to normal passwords. Curl supports |
| client-side certificates. All certificates are locked with a PIN-code, why |
| you need to enter the unlock-code before the certificate can be used by |
| curl. The PIN-code can be specified on the command line or if not, entered |
| interactively when curl queries for it. Use a certificate with curl on a |
| https server like: |
| |
| curl -E mycert.pem https://that.secure.server.com |
| |
| 12. REFERENCES |
| |
| RFC 2616 is a must to read if you want in-depth understanding of the HTTP |
| protocol. |
| |
| RFC 2396 explains the URL syntax |
| |
| RFC 2109 defines how cookies are supposed to work. |
| |
| http://www.openssl.org is the home of the OpenSSL project |
| |
| http://curl.haxx.se is the home of the cURL project |