Welcome to another tutorial, and once again another Visual Basic 5 application. Dutoon is pencil testing software for use by animators with video equipment. Might I suggest you also download the documentation as this provides many useful pieces of information.
The first thing we'll notice when we start this program, is that there appears to be no protection at all. But after about a minute a dialog box pops up with a machine specific code (which is in fact derived from your HD serial number). Now we could try >bpx rtcInputBox to snap SoftICE when the box appears, and then reverse the protection from there, this would indeed work but lets do some more analysis. When the nag box appears you'll note that the machine code is in fact 8 digits so lets try some responses of various lengths, you'll note how an 8 number response produces a request for you to restart.
In fact all the program has actually done is write your input to a text file (linetest.txt), and its actually possible to check the reading of this file by setting a >bpx CreateFileA and waiting for the program to read the file before it snaps the Input Box. However, lets just look around, remember how our message box appeared after 1 minute, indicative of some sort of timer?, timer.dll perhaps.
Disassembling this file will reveal some rather interestingly named export functions. Note how W32Dasm doesn't show these functions as actually being imported by linetest.exe (that'll take a while to disassemble so be warned). Several of these names should alert you immediately, (get_timer, init_timer, preview_initialize). So lets load the timer.dll exports into SoftICE and >bpx preview_initialize.
With the breakpoint enabled, run Dutoon and wait for the protection to reveal itself. After the break, press F11 and you should be looking at the following code.
:006D5C35 MOV [EBP-0134], EAX <-- EAX holds system
:006D5C3B CALL [MSVBVM50!__vbaSetSystemError]
:006D5C41 MOV EAX,[EBP-0134]
:006D5C47 CMP EAX,EBX <-- EBX was 0 here.
:006D5C49 MOV [EBP-1C],EAX
:006D5C4C JZ 006DF597 <-- This_jumps_rtcInputBox.
:006D5C52 CMP WORD PTR [ESI+000000A0],-01
:006D5C5A JNZ 006D5CA4 <-- This_will_hit_rtcInputBox_regardless.
Looking at this code warrants you to use a heuristic approach, keep in mind that we know that rtcInputBox is called at address 006D5DC4. A logical look at the disassembly indicates that forcing JZ 006DF597 is the only conceivable way to avoid the input box, and indeed if you make a patch to that effect the program will operate without the imposed limit.
:006D5C4C 0F 84 45 03 00 00 <-- JZ 006DF597.
:006D5C4C E9 46 03 00 00 90 <-- JMP 006D5F97 + NOP.
In fact we can perform a lot deeper analysis of this code, because it should be possible for us to patch timer.dll (thats where the real checking is done). Try bpx's on CreateFileA or GetVolumeInformationA in place of the timer exported functions and you'll be able to get a better picture of how this program actually works. Here's some alternative code you could patch (this time inside timer.dll).
:1000159B OR ESI,EDI <-- Logic operation.
:1000159D XOR ESI,A55AAA55 <-- Looks like the hard-coded in default XOR key.
:100015A3 CMP EAX,ESI <-- Compare HD serial number with XOR result.
:100015A5 JZ 10001666 <-- Jump_and_no_input_box.
So what you could also do is patch the XOR ESI,A55AAA55 to MOV ESI,EAX + relevant NOP's for an equal byte swap and the program would work just as well.
:1000159D 81 F6 55 AA 5A A5 <-- XOR ESI,A55AAA55.
:1000159D 8B F0 90 90 90 90 <-- MOV ESI,EAX + 4 x NOP's.