Ansoft Serenade v8.5 - Tutorial - (Full CD 152Mb's, Stripped Version 28Mb's).
v8.5 correction - Even I get it wrong sometimes..... ;-).
v8.7 addition - Latest tricks.

The protection of Ansoft Serenade v8.5 is pretty interesting for several reasons, the core strategy combines FLEXlm & Sentinel, integrity checks are thrown in just for good measure. For this tutorial we'll be using the FLEXlm v6.1 SDK, IDA v4.04 (get the FLAIR tools from Ilfak's site) & SoftICE, you should install Ansoft Serenade selecting local plug-in security module and license file (any license.dat will do as the installation performs no verification). I'll be focusing mainly on Ser85.exe which houses the protections I describe in the preamble, once this is understood the other components will quickly fall. A short recap is perhaps in order, FLEXlm is a licensing system based around 5 vendor keys, 2 encryption seeds and FEATURE names, earlier versions of FLEXlm were compromised by the inclusion of the license management dll (i.e. dynamic linking), RBS's FlexGen exploits these holes.

The key to FLEXlm is the recovery of the keys, seeds and FEATURE names, in theory once these are correct one only needs build lmcrypt and generate a valid license. The addition of the Sentinel key suggests that some patching is going to be required, however this isn't actually the case (thankyou 'flexible' FLEXlm) hardware key present or not, the license keys are all checked out using the same code. The first thing I recommend you do is get Ser85.exe inside IDA v4.04, from the /i86_n3 directory copy lmgr.lib and using pcf/sigmake (FLAIR tools) create a signature file. Use File/Load File/FLIRT Signature to apply the signature.

Looking firstly at the FLEXlm side you'll notice there is no longer a convenient lmgr32x.dll with export functions to bpx for, the routines are now statically linked to the executable although as Nolan Blender has pointed out, enough of the FLEXlm information remains, GlobeTrotter actually have a stripping utility for UNIX binaries but didn't think to port this to Windows targets. After applying the signature file IDA identifies a great deal of these functions, even the Sentinel API's too. I'll confess before I get to the real detail that I was pretty lazy so I wen't for fast recovery of the key information. I did some poking around in the code as I remembered from dan's essay that lc_new_job() randomizes the FLEXlm structure (in very early targets it was passed unencrypted to lc_init() ). As FLEXlm is heavily tied into Microsoft Visual C++ and this 'gronking' is performed using the system time a look for the Visual C++ time functions seems a good bet.

I also remembered that the checks for the default seed codes (shipped with the SDK) was a good place to fish the correct seeds, a simple disassembly search for '87654321' finds this code :-

:00429C4C CMP D, [EBP-34], 87654321 <-- Check for encryption_seed1.
:00429C53 JZ 00429C5E <-- Jump to Error.
:00429C55 CMP D, [EBP-30], 12345678 <-- Check for encryption_seed2.
:00429C5C JNZ 00429C85 <-- Good jump.
:00429C5E MOV EDI, FFFFFFA5 <-- Error Code (-91).

Routines inside Ser85.exe detect bpx type breakpoints and patching of key files, the checking code starts at 0040EFD3, here you'll see the names of the files that are checked and the rather obvious 'PUSH 7' instructions which produce a cryptic error message box asking you to call Ansoft for assistance. The files verified should give us a good idea where to look for other parts of the protection, Ansoft's developers evidently tried (as is good policy) to identify possible points of attack. Using bpmb style breakpoints we can quickly recover (what we think are valid) encryption_seed1 (0x7CB2B081) & encryption_seed2 (0x2DFE22B6).

Lets create a fake license.dat to work with :-

FEATURE feature1 vendorname 1.000 1-jan-0 0 00000000000000000000 / HOSTID=ANY ISSUER=CrackZ

Our first task is to recover the correct feature name(s), vendor name and version number, note that 1-jan-0 equates to a permanent license, the following 0 to uncounted and the 20 0's the license key. Run Ser85.exe and a helpful message box tells us that feature1 should really be SerenadeDesktop, after making the change run Ser85.exe again, this time the version information pops up, it should really be 8.000, you know the drill, make the change, run Ser85.exe again, this time we reach goal A, the license key doesn't check out, only our vendor name and license key remain to be found.

The function responsible for checking out licenses is still lc_checkout(), looking slowly through IDA's names you'll find it at 0041BFD2 (_lc_checkout). If you wan't to make tracing in SoftICE a touch more pleasurable generate a map file for use with the Loader (via msym). 2 breaks occur on _lc_checkout() (called from different locations), this is where things start to get interesting, the CALL at 0040FDF5 checks out a license for feature name 'f1' with version '1.0', this looks suspiciously like one of the examples in the FLEXlm documentation, and why if 'f1' was a legitimate feature would Ser85.exe report an error checking out feature 'SerenadeDesktop'.

Stepping back, reading through the FLEXlm manuals we note that lc_new_job() has to be called before any other FLEXlm functions, lc_new_job() also takes as one of its parameters a pointer to the VENDORCODE structure, the bpmb of course now goes on 0041F10A. Once again we get 2 breaks, the first at 0040FD1C, the second at 0040F9CE. The first break passes in via ECX a pointer to the VENDORCODE structure, when lc_new_job() completes the structure is filled (the vendor name 'compactd' can also be recovered here), the 2nd break does exactly the same (producing exactly the same structure), this suggests that feature 'f1' was probably some sort of internal test that made it to the final release, when we generate a valid license later we'll come back to this.

Lets retrieve the information from the VENDORCODE structure :-

  920DAE8F <-- encryption_seed1 ^ key5.
  C3413CB8 <-- encryption_seed2 ^ key5.
  4DC632DC <-- vendorkey1.
  70E7CE04 <-- vendorkey2.
  39C25E67 <-- vendorkey3.
  65E52191 <-- vendorkey4.

By simple XORing we can deduce that key5 = 0xEEBF1E0E. Everything seems to be over, insert the required information inside lm_code.h, build lmcrypt (-verfmt 4 -longkey switches) and generate a valid license file. This is exactly what I thought, however FLEXlm evidently has other ideas, lc_checkout() still fails with -5 (bad key). Lets take a second look at lc_new_job() and see what it does. As it turns out, in this target there is nothing really very sophisticated going on, vector call 0043A4C8 constructs the VENDORCODE structure using bytes from the .data section in a tedious fashion, this prevents combing of the .data section for likely looking keys, something I doubt anyone is going to try. Our old friend lc_init() is then called.

Lets trace inside lc_checkout(), the joys of yet another vector call await, this time its 0041C1BA. Rather than show lots of code I'll describe just the highlights. sub_426783 firstly checks the length of our version information (greater than 10 dec and your fired), _l_clear_error() calls memset() which clears 48 bytes of memory. _l_next_conf_or_marker() takes a pointer to the feature name and checks if it exists in the license file, if it does the return is non-zero. sub_41C68E performs several tasks, its main purpose is to check the license file, the version information is verified by _l_compare_version() and the expiration date by _l_extract_date(), with old style keys the 2nd/4rth/6th & 8th digits correspond to the expiry date, lmcrypt generates these correctly, further on _l_host() and ultimately _check_rainbow() access the hardware key if your license file had HOSTID=FLEXID=7-12345678 i.e. a Sentinel dongle with ID 12345678.

The Sentinel protection isn't really worthy of much discussion, sproFindFirstUnit(ID=0xB285) & sproRead(words 0 & 1) form its basis, sproFindNextUnit() should probably be patched to speed the routine up. sub_41CA41 is up next and call's 3 functions, _l_sg(), _l_extract_date() & sub_41D7B6. _l_sg() introduces yet another vector call, this time 0043AD1C, this gets the string length of the vendor name and creates a structure of random data (using the system time), this is the code you could have found by examining the _time() references.

_l_extract_date() gets the 2nd/4rth/6th & 8th digits of the license key and passes them to _l_ckout_crypt(). This is where everything gets checked out, its easy to see that if EAX is returned = 0 we are quickly fired with -5. With this in mind theres considerable tracing to be done (but be observant on the way) :-

:0041DE84 LEA ECX,[EBP-2420] <-- String data.
:0041DE8A PUSH ESI <-- License Key.
:0041DE8B SUB EAX,ECX <-- Gets length of String data.
:0041DE8D PUSH D, [EBP-10] <-- 0x289BEB8A.
:0041DE90 PUSH D, [EBP+14] <-- Gronked VENDORCODE structure.
:0041DE93 PUSH EAX <-- Push Length.
:0041DE94 PUSH ECX
:0041DE95 PUSH D, [EBP+08] <-- Random bytes structure.
:0041DE98 CALL sub_41E82E <-- Of course we want to look here.
:0041DE9D ADD ESP,18 <-- Adjust call stack.
:0041DEA0 MOV ESI,EAX <-- Return placed in ESI.
:0041DEA2 TEST ESI,ESI <-- Is ESI=0.
:0041DEA4 JNZ loc_41DEAA <-- Jump good FLEXlm license.
:0041DEA6 XOR EAX,EAX <-- -5.
:0041DEA8 JMP loc_41DEE8 <-- Jump to RET.

Just looking at these pushes you've got to be pretty confident we are close, the String data looks like so :-

D2 0F 37 AB 00 53 65 52 45 4E 41 44 45 44 45 53
 .  .  7  .  .  S  E  R  E  N  A  D  E  D  E  S

4B 54 4F 50 08 00 01 00 6A 61 6E 43 38 43 32 49
 K  T  O  P  .  .  .  .  j  a  n  C  8  C  2  I

53 53 55 45 52 3D 43 52 41 43 4B 5A
 S  S  U  E  R  =  C  R  A  C  K  Z

The first DWORD here is of great interest (check out the code at 0041E181), it must be investigated because it forms a critical part of the good key license generation, the rest of this data ought to look pretty familiar. At 0041EF8B we can recover the correct missing digits of the license key, note there are only 16 digits, the 4 digits which correspond to the expiration date are filtered out and are not affected by the encryption.

The first line of a valid license looks like so :-

FEATURE SerenadeDesktop compactd 8.000 1-jan-0 0 8CC8AC420F0F32F9713E HOSTID=ANY ISSUER=CrackZ

I promised earlier on to see what effect feature 'f1' has, it is checked out by exactly the same code described above, however having it validated crashes Ser85.exe with a memory access violation, there seems to be no good reason to investigate this any further. All that remains for the reader is to find the remaining feature names and the code that checks them out, this should prove a fairly simple task if you have understood everything that has gone before.

v8.5 correction

I'm sure you know the feeling I'm about to describe, you quickly find where a license gets generated and can't be bothered to find out why something that in theory should work, doesn't, yes I make mistakes too ;-). In v8.5 I managed to get lmcrypt to build successfully and generate licenses without errors (this made me think all my key information had to be correct), the licenses as you can read above, checked out bad, so I just assumed Ansoft had implemented something other than the default FLEXlm encryption, it didn't matter to me, I was able to recover valid license strings so I left it at that. This is not however the case, after reading Nolan Blenders Zendenc essay I realised I'd fallen victim to the 'FLEXlm hide the correct seeds trick', by following his example we can recover the correct key information and generate an lmcrypt that will give valid licenses.


In v8.7 Ansoft have once again implemented some changes. This version uses the v7.1b binaries, always remember to use the very helpful lmtools.exe if you are ever in doubt as to the version used. The vendor name has been changed from compactd to ansoftd and if we create a fake license to start working with now, we'll be fired with the error "SIGN= attribute required". The SIGN'ing option for FLEXlm has 2 modes of operation, one which uses a public key, the other not, fortunately for us, this version uses the simpler option.

The format of a license can therefore be :

FEATURE SerenadeDesktop ansoftd 8.700 permanent uncounted \
HOSTID=ANY SIGN=123412341234

Note here that the FLEXlm license key does NOT have to be present since it is the SIGN attribute that will decide whether the license is valid or not. Like the default FLEXlm license key before, the basic SIGN attribute is checked out using a similar side by side comparison, here is the code :

:004240D7 mov [ebp+var_18C], 8
:004240E1 cmp [ebp+arg_10], 66D8B337h <-- 2 lines very similar to default FLEXlm encryption.
:004240E8 jnz loc_4240F9
:00424287 mov al, byte_4869C8[edx]
:0042428D cmp ecx, eax
:0042428F jz loc_424295 <-- Lets check our SIGN= here.

Our task now is recovery of the correct key information for lmcrypt to generate correct licenses. By applying IDA signatures (ser87.exe) we can quickly deduce that l_sg() is sub_420B4A(), follow Nolan Blenders technique (see the Zendenc essay again). This works slightly different than expected since the job structure initially (before the call to lm_new() is filled with nulls), however after tracing over the routine the values will be filled in. At the end of this call we can feed the values to calcseed.exe.

vend+4 : 6B8F096C
vend+8 : 72AC5DA9
job+8 : CA448DA6
job+C : C1DEE000
job+10 : 9730C811

This gives us the real derived seeds of 7A8F4DCA & 63AC190F and the vendor name ansoftd.

As an aside you can verify that the sign attribute is formed using constants (depends on license features), part of the feature name (first 3 letters), the expiry and the encryption seeds, this is pretty similar to how the default FLEXlm licenses are made up ; one can also plug into lm_code.h the fake seeds 0x7EE1BBF8 & 0x67C2EF3D and a null pointer as the job structure and the license generated by lmcrypt will be checked out correctly. All that remains is to generate the license using the good seeds with lmcrypt.exe ;-).

As a further aside, recovery of the feature names in Serenade is hidden by a simple NOT/XOR encryption, looking at the linker arrangement this seems to be an additional security measure added by Ansoft themselves, not a feature of FLEXlm, the function takes as parameters a pointer to where to store the decrypted feature name and a pointer to the DWORD to cycle the encryption.

Return to FLEXlm Return to Entrance

© 1998, 1999, 2000, 2001 CrackZ. 30th December 2001.