% mechanize -- Hints

Hints for debugging programs that use mechanize.

Cookies
-------

A common mistake is to use `mechanize.urlopen()`, *and* the
`.extract_cookies()` and `.add_cookie_header()` methods on a cookie object
themselves.  If you use `mechanize.urlopen()` (or `OpenerDirector.open()`), the
module handles extraction and adding of cookies by itself, so you should not
call `.extract_cookies()` or `.add_cookie_header()`.

Are you sure the server is sending you any cookies in the first place?  Maybe
the server is keeping track of state in some other way (`HIDDEN` HTML form
entries (possibly in a separate page referenced by a frame), URL-encoded
session keys, IP address, HTTP `Referer` headers)?  Perhaps some embedded
script in the HTML is setting cookies (see below)?  Turn on
[logging](#logging).

When you `.save()` to or `.load()`/`.revert()` from a file, single-session
cookies will expire unless you explicitly request otherwise with the
`ignore_discard` argument.  This may be your problem if you find cookies are
going away after saving and loading.

~~~~{.python}
import mechanize
cj = mechanize.LWPCookieJar()
opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cj))
mechanize.install_opener(opener)
r = mechanize.urlopen("http://foobar.com/")
cj.save("/some/file", ignore_discard=True, ignore_expires=True)
~~~~

JavaScript code can set cookies; mechanize does not support this.  See [the
FAQ](faq.html#script).


General
-------

Enable [logging](#logging).

Sometimes, a server wants particular HTTP headers set to the values it expects.
For example, the `User-Agent` header may need to be [set](./doc.html#headers)
to a value like that of a popular browser.

Check that the browser is able to do manually what you're trying to achieve
programatically.  Make sure that what you do manually is *exactly* the same as
what you're trying to do from Python -- you may simply be hitting a server bug
that only gets revealed if you view pages in a particular order, for example.

Try comparing the headers and data that your program sends with those that a
browser sends.  Often this will give you the clue you need.  There are [browser
addons](faq.html#sniffing) available that allow you to see what the browser
sends and receives even if HTTPS is in use.

If nothing is obviously wrong with the requests your program is sending and
you're out of ideas, you can reliably locate the problem by copying the headers
that a browser sends, and then changing headers until your program stops
working again.  Temporarily switch to explicitly sending individual HTTP
headers (by calling `.add_header()`, or by using `httplib` directly).  Start by
sending exactly the headers that Firefox or IE send.  You may need to make sure
that a valid session ID is sent -- the one you got from your browser may no
longer be valid.  If that works, you can begin the tedious process of changing
your headers and data until they match what your original code was sending.
You should end up with a minimal set of changes.  If you think that reveals a
bug in mechanize, please [report it](support.html).


Logging
-------

To enable logging to stdout:

~~~~{.python}
import sys, logging
logger = logging.getLogger("mechanize")
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.DEBUG)
~~~~

You can reduce the amount of information shown by setting the level to
`logging.INFO` instead of `logging.DEBUG`, or by only enabling logging for one
of the following logger names instead of `"mechanize"`:

  * `"mechanize"`: Everything.

  * `"mechanize.cookies"`: Why particular cookies are accepted or rejected and why
they are or are not returned.  Requires logging enabled at the `DEBUG` level.

  * `"mechanize.http_responses"`: HTTP response body data.

  * `"mechanize.http_redirects"`: HTTP redirect information.


HTTP headers
------------

An example showing how to enable printing of HTTP headers to stdout, logging of
HTTP response bodies, and logging of information about redirections:

~~~~{.python}
import sys, logging
import mechanize

logger = logging.getLogger("mechanize")
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.DEBUG)

browser = mechanize.Browser()
browser.set_debug_http(True)
browser.set_debug_responses(True)
browser.set_debug_redirects(True)
response = browser.open("http://python.org/")
~~~~

Alternatively, you can examine request and response objects to see what's going
on.  Note that requests may involve "sub-requests" in cases such as
redirection, in which case you will not see everything that's going on just by
examining the original request and final response.  It's often useful to [use
the `.get_data()` method](./doc.html#seekable-responses) on responses during
debugging.

### Handlers ###

**This section is not relevant if you use `mechanize.Browser`.**

An example showing how to enable printing of HTTP headers to stdout, at the
`HTTPHandler` level:

~~~~{.python}
import mechanize
hh = mechanize.HTTPHandler()  # you might want HTTPSHandler, too
hh.set_http_debuglevel(1)
opener = mechanize.build_opener(hh)
response = opener.open(url)
~~~~

The following handlers are available:

**NOTE**: as well as having these handlers in your `OpenerDirector` (for
example, by passing them to `build_opener()`) you have to [turn on
logging](#logging) at the `INFO` level or lower in order to see any output.

`HTTPRedirectDebugProcessor`: logs information about redirections

`HTTPResponseDebugProcessor`: logs HTTP response bodies (including those that
are read during redirections)


<!-- Local Variables: -->
<!-- fill-column:79 -->
<!-- End: -->