Candee,
When running under a debugger it is possible to get different results particularly if the application has any uninitialised variables, or if it causes buffer ovverruns. If it is an overrun it would only cause an access violation if it just so happens that the overrun crosses a page boundary in memory and could easily differ depending on how it is run.
I would recommend that you check that the parameters passed into UTCOMP.dll are definitely valid and if you have the source for UTCOMPP.DLL take a look at how it handles those parameters.