If you mean VM86 mode as implemented by the I386 and higher
processors, you want James Turley's Advanced 80386
Programming Techniques. Unfortunately, it is out of
print. Fortunately, he has given me permission to scan it
in and put it on the web. Unfortunately, that's a slow
process, nowhere near completed.
If you mean to ask how a user-space program can use Linux's vm86() syscall, use "man vm86". You may find a use to modify a process's ldt, in which case you will want to read "man modify_ldt". Those man pages may be slightly obsolete -- check them against recent dosemu and/or Wine source code.
It seems clear to me from your description that your job should be relatively easy to do as a kernel device driver, for two reasons: