04-16-07 - 1

I've always wanted to make my apps clean Console/Windows apps. What I mean by that is - if I'm started from a console, I can either attach to that console or not, while if I'm started from an icon (no parent console) I can either pop up my own console or not. Turns out this is impossible because of the ass way Windows handles consoles and spawning processes. There is this nice function AttachConsole() which it seems like it would do the trick but basically it seems totally useless.

The goal is to have an app that is primarily a GUI app, but it has processing and help modes where you might want to just run it on the command line. For example, if someone runs "cbapp -?" and they do it from a cmd, I want to just output the help in the CLI, I hate it when apps pop up their usage in a little MessageBox. A classic example is the "msdev" or "devenv" app, which by default pops up the MS Visual Studio GUI interface, but you can run it from a CLI to do non-GUI builds of your projects and things like that. Basically I think pretty much every single app should be designed this way.

Anyway, there is a solution, but it's a bit ugly. First of all, you compile your whole app twice, once for CONSOLE and once for WINDOWS subsystems. The two apps are in the same directory and have the same name, but the console one is named ".com" ; because of the cmd execution order rules if you run the app from the command like you will run the .com version, obviously shortcuts point at the .exe version. Now, the .com version will attach to the console - if you decide it's in a mode where it should be a detached windows app you have it spawn the .exe version. Conversely, if the .exe is run and you decide you need a console, you can just call :

if ( AllocConsole() ) // does nothing & returns false if we already have one
	freopen("CONIN$","rb",stdin);   // reopen stdin handle as console window input
	freopen("CONOUT$","wb",stdout);  // reopen stout handle as console window output
	freopen("CONOUT$","wb",stderr); // reopen stderr handle as console window output

Which puts a console on your Windows app. Apparently several MS apps use this .com/.exe trick.

Now, there is one last problem that I hate. The console window you pop has a close box on it, and this close box has a nasty property - when it's clicked your app is *terminated* like a KillProcess() - you don't run any shutdown code! This is pretty evil and retarded IMO. I haven't found my ideal solution to this but the two things that come to mind are : 1) hook/hammer that window to get rid of its close box, or 2) use one of the many Windows console replacement libraries (eg. don't use the built-in windows console at all). You could also write your own console pretty easily, especially if you're only doing output. You just direct stdout to a windows file and wait on it in a thread to update a little text pane window.

Part of the weirdness with the console window is that even if your app makes the console you don't own it. All console windows in Windows are owned by "csrss.exe" which an always-running process which acts as the console server. When you call AllocConsole() it actually sends a message to csrss and tells it to make the console window; it also makes some memory mapped files and gives you the handles to them which are then your stdin/stdout/stderr. To talk to the console you write those files, it sees they changed and update the window. This is what makes the console totally async from your app and means you app can crash and still have a valid console, which is what makes it nice for debugging. The csrss thing sort of explains why you just get a KillProcess. When the close box on the console is hit, csrss gets that message, sees the PID that's attached to that console and just kills it.

Addendum :

Actually it can be easier than this. You don't need to do the whole .com/.exe thing unless you want your app to be significantly different in windows & console mode. Instead you can just do this easy thing :

Make your app a console app with a normal main() entry point. When your app decides that it wants to detach and be a windows app without a console, you just respawn yourself like this :

	// I'm consoled and I don't want to be
	fprintf(stderr,"Respawning detached\n");

	STARTUPINFO si = { 0 };
	PROCESS_INFORMATION procinfo = { 0 };
	si.cb = sizeof(si);

	BOOL ok = CreateProcess(
		FALSE, // 	bInheritHandles
		NULL, // = inherit environment
		NULL, // = inherit curdir

	if ( ! ok )
		fprintf(stderr,"ERROR : CreateProcess failed !\n");


This will relaunch your app separated from the parent handles (so if you're run from cmd you don't inherit its stdin/stdout). Your app is still a console app, it just won't have a console and the stdin/stdout will be null files. You can always pop your own console in the future too.

Actually there is one ass crappy thing about this - if you run this from a shortcut it first starts as a console app, a console will pop up, then you'll respawn and the console dies, so you have this evil flash at startup. So, you can solve that by going the .com/.exe route, and I still don't have a happy solution to this problem. If you do the proper thing and use the .com/.exe trick then your .com just spawns the .exe using the above code, (but you have to change the app name in the commandline).

Another semi-related tip you might not know : there's really no point in defining _CONSOLE or whatever in your project, and in the linker you can set subsystem=NOT SET. Then MSVC will just autodetect whether you implemented main() or WinMain() and do the right thing.

More addendum : Tom points out you can fix the console-making-your-app-terminate thing. You just implement a ConsoleControlHandler and catch CTRL_CLOSE_EVENT. I have no idea why Windoze didn't make the default version of that raise a signal, which would've made all the ANSI apps work nice, but they didn't. The only funny bit is that a ConsoleControlHandler is run on a little thread that's created in your app just to run it (presumably with CreateRemoteThread from the csrss). That means you can't just PostQuitMessage() because you're not on the right thread, and you also can't just PostMessage() to your main window unless you're a bit careful (okay, in practice you probably can do that 99% of the time). Anyway, a safe thing to do is like this :

volatile LONG g_consoleClose = 0;

BOOL MyConsoleCtrlHandler(DWORD fdwCtrlType) 
	// note : this is run on a little thread inside my process which is created just to run this
	//	that allows me to pop a message box or something if I want
    switch (fdwCtrlType) 
        // Handle the CTRL+C signal. 
        // CTRL+CLOSE: confirm that the user wants to exit. 
        case CTRL_C_EVENT: 
        case CTRL_CLOSE_EVENT: 

            return TRUE; 
            return FALSE; 

And then in your main thread you do this :

		if ( InterlockedExchange(&g_consoleClose,0) > 0 )

Or whatever you want to do in there to signal your app its time to get out.

No comments:

old rants