So you want your WordPress site to allow you to have a different 404 error page for a subdirectory.  These instructions are for WordPress sites running on Apache.

The issue with adding custom 404 error pages to your WordPress site is due to WordPress using Apache’s rewrite on your entire site.  In order to prevent this from happening on your subdirectory, you need to add a .htaccess file with the following code:

<IfModule mod_rewrite.c>
RewriteEngine Off
</IfModule>

ErrorDocument 404 /subdirectory/404.html

This code prevents URL rewrites on your subdirectory, and specifies your custom 404 error page.  The 404 error page should link to the custom file you want to display to your visitors.

Here’s a link to some stylish 404 error pages.
http://dzineblog.com/2008/11/custom-error-404-pages.html

If you’re still having trouble with getting your custom error page to display, make sure that your subdirectoy has the AllowOverride permission set in your Apache configuration.  In your Apache configuration, you can set AllowOverride on that directory with this code:

<Directory “/path/to/your/wordpress/subdirectory”>
AllowOverride All
</Directory>