Assembling IDA output with MASM - 'file exceeds line limits' and 'instruction prefix not allowed'

Question

When compiling the exported assembly from IDA with MASM, I get the following two errors:

test.exe.asm(32767) : fatal error A1019: invalid debug and browser data; file exceeds line limit

test.exe.asm(2697) : error A2068: instruction prefix not allowed

Here is the code that triggers the last error:

mov     eax, ds:dword_40D028
test    eax, eax
jz      short loc_401D40
rep retn                     <----- Line 2697

First of all: How can those errors occure? The original .exe is working fine. Obviously the disassembly cannot produce the exact original assembly but it should at least produce working code, shouldn't it?

Secondly: How do I fix this?

Edit: When trying to compile the assembly with JWASM I no longer get the exceeding line limit error but the following errors:

test.exe.asm(40764) : Error A2150: Missing operator in expression
test.exe.asm(40767) : Error A2150: Missing operator in expression
test.exe.asm(40768) : Error A2150: Missing operator in expression
test.exe.asm(40769) : Error A2150: Missing operator in expression
test.exe.asm(40772) : Error A2150: Missing operator in expression
test.exe.asm(40773) : Error A2150: Missing operator in expression
test.exe.asm(40774) : Error A2150: Missing operator in expression
test.exe.asm(40777) : Error A2150: Missing operator in expression
test.exe.asm(40778) : Error A2150: Missing operator in expression
test.exe.asm(40779) : Error A2150: Missing operator in expression
test.exe.asm(40782) : Error A2150: Missing operator in expression
test.exe.asm(40783) : Error A2150: Missing operator in expression
test.exe.asm(40815) : Error A2150: Missing operator in expression
test.exe.asm(40816) : Error A2150: Missing operator in expression
test.exe.asm(40817) : Error A2150: Missing operator in expression
test.exe.asm(40818) : Error A2150: Missing operator in expression
test.exe.asm(40819) : Error A2150: Missing operator in expression
test.exe.asm(40820) : Error A2150: Missing operator in expression
test.exe.asm(40821) : Error A2150: Missing operator in expression
test.exe.asm(40822) : Error A2150: Missing operator in expression
test.exe.asm(40823) : Error A2150: Missing operator in expression
test.exe.asm(40824) : Error A2150: Missing operator in expression
test.exe.asm(40825) : Error A2150: Missing operator in expression
test.exe.asm(40826) : Error A2150: Missing operator in expression
test.exe.asm(40827) : Error A2150: Missing operator in expression
test.exe.asm(40828) : Error A2150: Missing operator in expression
test.exe.asm(40829) : Error A2150: Missing operator in expression
test.exe.asm(40830) : Error A2150: Missing operator in expression
test.exe.asm(40831) : Error A2150: Missing operator in expression
test.exe.asm(40832) : Error A2150: Missing operator in expression

Here is the code beginning from line 40759 (including comments)

; Segment type: Pure data                        <----- Line 40759
; Segment permissions: Read
imports         segment para public 'DATA' use32
                assume cs:imports
                ;org 40E000h
__IMPORT_DESCRIPTOR_kernel32 dd rva off_40E098 ; Import Name Table
                dd 0                    ; Time stamp
                dd 0                    ; Forwarder Chain
                dd rva aKernel32Dll     ; DLL Name
                dd rva AllocConsole     ; Import Address Table
__IMPORT_DESCRIPTOR_user32 dd rva byte_40E12C+4 ; Import Name Table
                dd 0                    ; Time stamp
                dd 0                    ; Forwarder Chain
                dd rva aUser32Dll       ; DLL Name
                dd rva MessageBoxA      ; Import Address Table
__IMPORT_DESCRIPTOR_msvcrt dd rva byte_40E13C+4 ; Import Name Table
                dd 0                    ; Time stamp
                dd 0                    ; Forwarder Chain
                dd rva aMsvcrtDll       ; DLL Name
                dd rva printf           ; Import Address Table
__IMPORT_DESCRIPTOR_Advapi32 dd rva byte_40E154+4 ; Import Name Table
                dd 0                    ; Time stamp
                dd 0                    ; Forwarder Chain
                dd rva aAdvapi32Dll     ; DLL Name
                dd rva CryptAcquireContextA ; Import Address Table
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
                db    0
aKernel32Dll    db 'kernel32.dll',0     ; DATA XREF: imports:0040E00C\u2191o
                align 2
aUser32Dll      db 'user32.dll',0       ; DATA XREF: imports:0040E020\u2191o
                align 2
aMsvcrtDll      db 'msvcrt.dll',0       ; DATA XREF: imports:0040E034\u2191o
                align 2
aAdvapi32Dll    db 'Advapi32.dll',0     ; DATA XREF: imports:0040E048\u2191o
                align 4
;
; Import names for kernel32.dll
;
off_40E098      dd rva word_40E170      ; DATA XREF: imports:__IMPORT_DESCRIPTOR_kernel32\u2191o
                dd rva unk_40E180
                dd rva word_40E18E
                dd rva word_40E19E
                dd rva word_40E1AE
                dd rva word_40E1C4
                dd rva word_40E1D6
                dd rva word_40E1E6
                dd rva word_40E1F6
                dd rva word_40E20A
                dd rva word_40E21E
                dd rva unk_40E230
                dd rva unk_40E244
                dd rva word_40E256
                dd rva word_40E26C
                dd rva word_40E280
                dd rva word_40E28E
                dd rva word_40E29E
                dd 0
imports         ends

In addition to that another error also occurred:

test.exe.asm(40884) : Error A2209: Syntax error: :
test.exe.asm(40890) : Error A2209: Syntax error: :
test.exe.asm(40897) : Error A2209: Syntax error: :

The code beginning from line 40882:

                extrn LoadLibraryA:dword <------ line 40882
                extrn CreateThread:dword
                extrn byte_40E12C:byte:0Ch
                                        ; DATA XREF: imports:__IMPORT_DESCRIPTOR_user32\u2191o
;
; Imports from user32.dll
;
                extrn MessageBoxA:dword ; DATA XREF: imports:0040E024\u2191o
                extrn byte_40E13C:byte:10h
                                        ; DATA XREF: imports:__IMPORT_DESCRIPTOR_msvcrt\u2191o
;
; Imports from msvcrt.dll
;
                extrn printf:dword      ; DATA XREF: imports:0040E038\u2191o
                extrn memset:dword
                extrn byte_40E154:byte:10h

Use a better assembler. If you insist on your version of masm, maybe split the file into parts as the first error seems to complain about it containing too many lines. As for the second error, encode the instruction manually or if you don't particularly care just use `retn` without the `rep` prefix. — Jester, May 23 '21 at 01:23
@Jester I don't really care about the assembler. It's just that IDA uses MASM format for their disassembler so I am kind of forced to use MASM. But If you know how I could change the IDA disassembly to a better assembler than that would also help me. Also: What do you mean by "encode the instruction manually"? — abcdefghi019283, May 23 '21 at 01:29
`db 0xf3, 0xc3` or similar. "Better" in this context means maybe a newer version of masm that doesn't have these problems or some other assembler that still uses masm syntax. — Jester, May 23 '21 at 01:32
@Jester Do you know a version that is new enough but can still work on it's own? I don't want to install Visual Studio just to compile the assembly put out by IDA. — abcdefghi019283, May 23 '21 at 01:43
That's a very unusual file name. Usually you'd expect test.asm for the source file and test.exe for the executable. The fact that they're combined like that seems odd. — David Wohlferd, May 23 '21 at 01:52
Does JWASM work on your code? (And BTW, `rep ret` is [only useful as a workaround for an AMD K10 branch-prediction limitation](https://stackoverflow.com/a/32347393/224132). GCC's default `-mtune=generic` used it by default for years (until GCC8) in `ret` instructions that could run as the next instruction after a branch (either taken or not taken), but it's not helpful for any other CPUs. Except for making instructions longer for alignment.) So if you're reworking the code, you probably want to omit the `rep` in front of ret, as long as there are still `align` directives on later labels. — Peter Cordes, May 23 '21 at 01:53
@DavidWohlferd: from comments, it's apparently IDA disassembly output, which I guess explains the name. — Peter Cordes, May 23 '21 at 01:53
Can you get IDA to disassemble into different syntax, like GAS or NASM? If not, you could use Agner Fog's `objconv` disassembler (https://www.agner.org/optimize/#objconv) to make source for good Free-software assemblers like NASM that are easy to install as stand-alone. — Peter Cordes, May 23 '21 at 01:56
@PeterCordes I used JWASM and it no longer gives me the "line exceeding error". But now I got new errors that I have added to the opening post. — abcdefghi019283, May 23 '21 at 02:18
@PeterCordes I don't know how to do that in IDA but I will try out the other disassembler and update my post. — abcdefghi019283, May 23 '21 at 02:19
So which lines are the error lines? The ones like `__IMPORT_DESCRIPTOR_kernel32 dd rva off_40E098`? Oh well, I guess JWASM doesn't know the `rva` keyword, whatever it does in MASM. Whatever those blocks are doing, maybe JWASM does the whole DLL import thing differently, and you can replace that with a .lib file or something? I don't know how Windows stuff works. — Peter Cordes, May 23 '21 at 02:22
@PeterCordes The thing is that I plan to disassemble the binary using IDA, make some (functional) changes to the assembly and recompile it. I thought that the assembly would be still intact and usable when exported from IDA but this does not seem to be the case. Do you know a way of converting a pe32 to assembly without corrupting the assembly? — abcdefghi019283, May 23 '21 at 02:41
It probably is usable with modern MASM; I wouldn't call it "corrupting". It's not IDA's fault you're trying to assemble it with a crappy old version with file-size limits too small to handle the executable you disassembled. (It arguably is IDA's fault for not disassembling into a format that can be handled without other commercial software like MASM, though.) IDK if newer MASM will accept `rep ret`, but that's certainly how a disassembler *should* represent that machine code. — Peter Cordes, May 23 '21 at 02:50
If `objconv` works for disassemble / edit / reassemble, I'd recommend that. It's free and open source, and can disassemble into NASM, GAS, or MASM formats. You'll have to verify that library function imports still work when you reassemble, but if so then you don't need IDA. (Note that most people modifying executables don't disassemble/reassemble the whole thing; they add a new section to the binary so it doesn't have to be re-linked, and patch a few places to jump there. A "code cave" is the term to search for.) — Peter Cordes, May 23 '21 at 02:52
@PeterCordes Do you have a recommendation for other disassembler and assembly compiler combination that does not lead to those errors? My main goal is to convert a pe32 to assembly and back, without errors occurring. I have not much experience when it comes to disassembling, so I can switch to whatever you recommend. — abcdefghi019283, May 23 '21 at 02:58
IDA pro output wasn't designed to be loaded by MASM directly. it does need fixing. For example `Error A2150: Missing operator in expression` is because MASM doesn't know what to do with `rva` . RVA isn't a keyword in MASM. — Michael Petch, May 23 '21 at 03:13
MASM doesn't support `extrn byte_40E12C:byte:0Ch` the problem is the `:0ch` on the end — Michael Petch, May 23 '21 at 03:19
@abcdefghi019283: Yes, for the 3rd time, `objconv` disassembler, NASM assembler would be what I'd try first. Have you even *tried* that? You just keep asking me for a recommendation after I've already made one, without saying if it worked. (It might not be 100%; e.g. for Linux ELF files, I think it might mess up the PLT in a file that had one, because it won't use `push strict dword` to make each block a fixed length. So you might need to do some text processing on library function references there or for Windows PE32 files. Normally you let an assembler + linker take care of the details.) — Peter Cordes, May 23 '21 at 03:23
@MichaelPetch: Note that A2150 error from is from JWASM, which the querent also tried. I assume you're correct that modern MASM would also have the same problems with both those things you mentioned, though. — Peter Cordes, May 23 '21 at 03:26
@PeterCordes : JWASM/MASM/WASM do not have a keyword RVA. Period. End of story. They'll all generate some kind of error processing that line. The size on EXTRN is not supported by any of those assemblers either. I have said that IDA PRO was not meant to be run through MASM (and there aren't any variants that will do it either) without modification. This is something IDA users know. You aren't going to find a version of the assembler that will assemble this without changes. — Michael Petch, May 23 '21 at 03:50
@MichaelPetch. You are correct. Neither do Ghidra, BinaryNinja and many other reverser's tools produce disassembly listings from non-trivial executables which can be used by an assembler without modification. — fpmurphy, May 23 '21 at 06:10

Assembling IDA output with MASM - 'file exceeds line limits' and 'instruction prefix not allowed'

0 Answers0