Thursday, October 18, 2007

Hudsuckr: Advanced Windows proxy configuration from the command line

Windows stores proxy configuration settings in the registry, in HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings. However, the settings there like "ProxyServer" "ProxyEnable" can be somewhat misleading, and certain advanced settings (like the setting to "Automatically detect settings" using WPAD) aren't available as separate registry keys.

In IE4, proxy settings were stored in the "Internet Settings" key as simple strings (REG_SZ) and ints (REG_DWORD), but since IE5 it has been possible to set proxy configurations separately for each Internet connection. For example, you could set up your computer to use a proxy when you're using a VPN, use WPAD when you're on dial-up, and to go directly to the Internet when you're corrected directly using the LAN. These "per connections" settings are stored in the "Internet Settings\Connections" key; unfortunately, they're stored there in REG_BINARY blobs that are not meant to be edited directly. (When you use the "Internet Options" Control Panel, it automatically updates both the REG_BINARY blobs and the regular registry settings in "Internet Settings" automatically.)

That makes it difficult to configure "per connection" settings (like WPAD) using command line tools, including proxycfg.exe and others. We needed to be able to automatically enable/disable WPAD in Selenium RC, so I've coded up a simple command-line executable called "hudsuckr" that lets you do it. (It's named after the classic 1994 Coen brothers movie The Hudsucker Proxy. "Hudsuckr" contains no "e" because that makes it easier to Google and because it makes it more Web 2.0. ;-)

Here's the usage documentation that you get when you run "hudsuckr --help" from the command line:

    Hudsuckr Windows Proxy Configuration Tool: "You know, for kids!"
    
    Windows manages Internet proxy connection information in the registry;
    each Internet "connection" can have its own separate proxy
    configuration.  These settings correspond to settings in the "Internet
    Options" Control Panel, under the "Connections" tab.
    
    Run "hudsuckr" without arguments to print out the current proxy
    configuration details.  We print out the name of the current active
    connection, the four connection flags (DIRECT, PROXY, AUTO_PROXY_URL,
    AUTO_DETECT), and three strings: PROXY_SERVER, PROXY_BYPASS, and
    AUTOCONFIG_URL. The seven settings are described in MS documentation
    available here: http://msdn2.microsoft.com/en-us/library/aa385145.aspx
    
    Run "hudsuckr" with exactly eight arguments to set the proxy
    configuration, like this:
      hudsuckr (null) true true true true "localhost:4444" "" "file://c:/proxy.pac"
    
    Specify the name of the connection first (or use the LAN settings by
    specifying "(null)"), then set the four flags using "true" and
    "false", then the proxy server (with a colon to specify the port), the
    list of servers to bypass delimited by semi-colons (with "" as
    a special string that bypasses local addresses), and finally the URL
    to a proxy PAC autoconfiguration file.  Use "" or "(null)" to leave
    string settings blank/empty.
    
    If you're still confused about the flags, look at the proxy settings
    in the "Internet Options" Control Panel. See how you can check those
    checkboxes independently of one another? The flags correspond to those
    checkboxes.  If AUTO_DETECT is true, IE will try to use WPAD; if
    successful, WPAD will override the specified AUTOCONFIG_URL. If an
    AUTOCONFIG_URL is detected (by AUTO_DETECT) or specified directly
    (AUTO_PROXY_URL is enabled), IE will use the autoconfig script as a
    proxy PAC file.  If no AUTOCONFIG_URL was specified or detected, IE
    will attempt to use the server specified in PROXY_SERVER if the PROXY
    flag is enabled; it will bypass the proxy for the list of servers
    specified in PROXY_BYPASS. Finally, if PROXY, AUTO_DETECT and
    AUTO_PROXY_URL are all set to false, IE will attempt to contact the
    web server directly.  Note that the DIRECT flag always appears to be
    true, even if the PROXY flag is true; we recommend you leave it that
    way, too.

Here's what you might see if you run hudsuckr without command line arguments:

    > hudsuckr.exe
    ACTIVE_CONNECTION=(null)
    PROXY_TYPE_DIRECT=true
    PROXY_TYPE_PROXY=false
    PROXY_TYPE_AUTO_PROXY_URL=false
    PROXY_TYPE_AUTO_DETECT=false
    INTERNET_PER_CONN_PROXY_SERVER=(null)
    INTERNET_PER_CONN_PROXY_BYPASS=(null)
    INTERNET_PER_CONN_AUTOCONFIG_URL=(null)

Or, after you've configured lots of proxy settings:

    > hudsuckr.exe (null) true true true true "localhost:4444" "" "file://c:/proxy.pac"
    ACTIVE_CONNECTION=(null)
    PROXY_TYPE_DIRECT=true
    PROXY_TYPE_PROXY=true
    PROXY_TYPE_AUTO_PROXY_URL=true
    PROXY_TYPE_AUTO_DETECT=true
    INTERNET_PER_CONN_PROXY_SERVER=localhost:4444
    INTERNET_PER_CONN_PROXY_BYPASS=
    INTERNET_PER_CONN_AUTOCONFIG_URL=file://c:/proxy.pac

To turn off all proxies for the LAN connection, you might run a command like this:

    > hudsuckr (null) true false false false (null) (null) (null)
    ACTIVE_CONNECTION=(null)
    PROXY_TYPE_DIRECT=true
    PROXY_TYPE_PROXY=false
    PROXY_TYPE_AUTO_PROXY_URL=false
    PROXY_TYPE_AUTO_DETECT=false
    INTERNET_PER_CONN_PROXY_SERVER=(null)
    INTERNET_PER_CONN_PROXY_BYPASS=(null)
    INTERNET_PER_CONN_AUTOCONFIG_URL=(null)

Download hudsuckr.exe
Hudsuckr source code

Friday, September 07, 2007

Windows, killableprocess, and the Case of the Suicidal Parent (or: How to kill a process and all of its subprocesses reliably on Windows)

The two major browsers available on the market today (Firefox and IE) are sometimes "suicidal parents" when launched from the command line. That is, they occasionally have to re-launch themselves, by spawning a new copy of their executable (firefox.exe or iexplore.exe) and killing the currently running process. In these cases, the orphaned child process lives on to become the GUI that you actually see on the screen.

Firefox commits suicide when it sees that it needs to make significant adjustments to its user profile directory, e.g. when it has detected new versions of your extensions/add-ons. IE commits suicide on Vista when running in Vista's new "Protected" mode. When you launch iexplore.exe from the command line on Vista in Protected mode, your iexplore.exe process spawns a copy of "ieuser.exe" (if one isn't running already) and allows THAT process to launch a new copy of iexplore.exe.

This suicidal behavior is invisible to normal end users, but it can be very difficult for people who want to automatically launch and kill browsers for automated testing, like I do. For example, if you spawn a suicidal parent in Java and then try to .destroy it, you'll run straight into Java bug 4770092 (Process.destroy does not kill multiple child processes).

Benjamin Smedberg posted an interesting blog entry about a way to use Python to handle child processes: killableprocess.py. On Windows, killableprocess.py handles child processes by launching them inside of a Windows "Job" object. Processes within a Job may be assigned special security privileges or restrictions, and, by default, whenever any process in a Job creates a subprocess, the subprocess becomes a member of the same Job. (Job objects were introduced in Windows 2000.) It's easy and convenient to terminate an entire Job at once, killing all processes within it, even if the processes are orphans of suicidal parents.

Using Jobs to manage a process tree is a great idea, but not everybody in the world has the Python runtime installed, so I decided to code up something like it in C++.

The compiled executable (killableprocess.exe) can be used to launch any other process you like, assigning that process to a new unnamed Job. killableprocess.exe will automatically terminate if every process in the Job has died. It's also designed to handle shutdown signals (Ctrl-C and Ctrl-Break), to terminate the Job as killableprocess.exe is being shutdown. As a fallback, it additionally registers a "job limit" with the operating system to automatically kill the Job if killableprocess.exe is forcibly terminated. (Beware... the job limit doesn't work if you use Process Explorer.)

You can use killableprocess.exe as a wrapper around other processes that you expect to be suicidal parents.

Download killableprocess.exe
Source code for killableprocess.exe

Tuesday, May 08, 2007

"Buildable" JDK doesn't build on Windows: missing t2k.lib

What a mess.

This morning at JavaOne, Rich Green announced that you could at long last build a working JDK/JRE from GPL'd source... mostly. A few key libraries remained non-GPL'd (encumbered) so they had to be shipped as binary plugs. That's fine... we can work on that. If we can build the code, that is.

But if you go to openjdk.java.net right now, and go download openjdk-7-ea-src-b12-06_may_2007.zip and jdk-7-ea-plug-b12-windows-i586-06_may_2007.jar, you'll find that the JDK won't build correctly on Windows. Instead, you'll get this error message when you run "make sanity":

ERROR: Can't locate t2k import library.
       Please check your access to
           /c/devtools/jdk-7-binary-plugs/jre/bin/t2k.lib
       and/or check your value of ALT_CLOSED_LIB_DIR.

That file is indeed missing from the Windows binary plugs. We have a t2k.dll, t2k.pdb, and t2k.map, but no t2k.lib. At JavaOne today Dmitri Trembovetski confirmed that this was a real issue, and that t2k.lib is a core part of the graphics rasterizer, one of the key encumbered components of the binary plugs.

Dmitri also suggested a workaround, which I'm not too excited about but which would probably work: you could, if you wanted, go get the JRL JDK 7, accept the JRL, and build a t2k.lib from that source. I'm concerned about my contaminating the new GPL'd Java by doing that, so I think I don't want to do that. Instead, someone at Sun should probably release another version of the binary plugs archive with the necessary file(s) included.

Dmitri and Tom Marble suggested that I file a bug on bugs.sun.com, but bugs filed there aren't immediately visible via the web. I was not even assigned a bug number, so I can't reference it here. And, unfortunately, even though bugs.sun.com HAS a "build" category for J2SE bugs, ordinary users can't file bugs against it. I had to file the bug against "installation" instead.