I've been unable to run VM/ESA under Hercules Hyperion. A guy nicknamed ~pjfarley3 seems to have succeeded, however. As for me, I didn't try for long. But, hey: I got the P/390 DASD's and a working z/VM instance, so might as well make good use of it, after being able to successfully running Linux on that instance — and it's running besides VM/ESA, right now.

To make it work, there are some important stuff to do. You can create an account similar to that created for Linux, albeit you surely won't need 1G of RAM. 256 MB ought be enough for everyone, here.

Also, you might want to change one thing or another. Here's the directory entry for my VM/ESA user:

USER VMESA ADUBO99 256M 512M G                                          
   IPL CMS                                                              
   MACHINE ESA 4                                                        
   DEDICATE 0123 0223                                                   
   DEDICATE 0124 0224                                                   
   DEDICATE 0125 0225                                                   
   DEDICATE 0126 0226                                                   
   DEDICATE 0128 0228                                                   
   DEDICATE 0700 0700
   DEDICATE 0701 0701
   CONSOLE 0200 3270 T                         
   SPOOL 000C 2540 READER *                                             
   SPOOL 000D 2540 PUNCH A                                              
   SPOOL 000E 1403 A                                                    
   LINK MAINT 0190 0190 RR                                              
   LINK MAINT 019D 019D RR                                              
   LINK MAINT 019E 019E RR                                              
   MDISK 0191 3390 4439 100 LINUX0 MR                                   

What's important here is: console is at 0200 and set as 3270 (though it won't set the TERM MODE, forcing you to do it later and thus crashing CMS). The last two dedicated devices (0700-701) are CTCA/LCS devices added to Hercules. It'll be explained next. And the other dedicated devices are DASD's. I had also a NIC defined here, but removed it because VM/ESA (V2R4) lacks QDIO support.

Like I said, it doesn't support QDIO. Neither any kind of modern networking stuff. It's a rather ancient OS. So, you get two options: LCS and CTC. CTC is rather painfull to configure, IMO. As far as I know, if you want the guest to communicate with the host (z/VM), you should configure z's TCP/IP stack to handle a CTC endpoint… And I have no patience to debug PROXYARP bugness. If it works for you, though, congratulations.

What worked for me was to add a LCS interface to Hercules. It's layer 2, so it can be bridged to the rest of my home network. Then, z/VM attaches those devices to VMESA user. On my Linux host it surfaces as tap0, which is bridged with my secondary network card and the VSW2 interface (mainfr1). I won't go into details on my network setup, though, because it's rather clusterfucky.

Refer to the end of the document for VM/ESA's PROFILE TCPIP. It's not entirely correct. But works enough to let me login via Telnet. It was a minimum viable product derived from the original Token Ring LCS's config. I didn't mess much because XEDIT is absolutely annoying. I'm sure that some dinosaurs might like it better than a decent editor, like VIM, though.

If you ever end up with a Disabled Wait ended with 1010 code, it's probably because CMS seems to force terminal into 3215 mode. Problem is: VM/ESA expects system operator console to be at the correct position (0200) and of 3270 type. So, just run that command:

TERM CON 3270

Which, probably, will crash CMS. But the guest will work as intended.

Once you're done working on VM/ESA, you might as well leave your z/VM session. But that will probably cause your guest to be terminated. To avoid that, before IPL'ing the system, run:

SET RUN ON

Might be a good idea to put it into PROFILE EXEC.

As per that thread, do as follows (René Ferland's post):

logon maint
cprelease a
link maint cf1 cf1 mr
access cf1 z
xedit system config z
access 193 x
cpsyntax system config z
release z
link maint cf1 cf1 rr
cpaccess maint cf1 a r
logoff

VM/ESA: PROFILE TCPIP

; ======================================================================
; File Name:   PROFILE TCPIP
; Description: Sample P/390 TCP/IP Configuration File (SYS1: 9.67.174.1)
;
;  This configuration file is used to define system operation
;  parameters, Telnet and network configuration information for your
;  TCP/IP environment.  The definitions within this file are used by
;  the TCPIP virtual machine to establish your TCP/IP services during
;  initialization, although some statements within this file can be
;  changed dynamically by using the OBEYFILE command.
;
;  For detailed information about the use and syntax of configuration
;  statements used within this file, see "Chapter 6. Configuring the
;  TCPIP Virtual Machine" in the "IBM TCP/IP for VM Planning and
;  Customization" manual.
;
;  Note: Comments within this file must begin with a semicolon (;)
;        followed by at least one blank.
;
; ======================================================================

; ======================================================================
; Free Pool Statements
;
; Use the statements below to define the allocation of control blocks
; and data buffers used by the TCP/IP stack (TCPIP) and other TCP/IP
; service virtual machines.  Ensure the Free Pool statements precede
; all other statements within this file.
; ======================================================================

ACBPOOLSIZE                 1000
ADDRESSTRANSLATIONPOOLSIZE  1500
CCBPOOLSIZE                 150
DATABUFFERPOOLSIZE          160  8192
ENVELOPEPOOLSIZE            750
IPROUTEPOOLSIZE             600
LARGEENVELOPEPOOLSIZE       50   8192
RCBPOOLSIZE                 50
SCBPOOLSIZE                 256
SKCBPOOLSIZE                256
SMALLDATABUFFERPOOLSIZE     0
TCBPOOLSIZE                 256
TINYDATABUFFERPOOLSIZE      0
UCBPOOLSIZE                 100

; ----------------------------------------------------------------------
; Monitorrecords statement
;
; Use the MONITORRECORDS statement to select the monitor data records
; that are produced by TCPIP, if any.
; ----------------------------------------------------------------------
MONITORRECORDS

; ----------------------------------------------------------------------
; Tracing Statements
;
; Define tracing statements below, to log events for diagnostic
; purposes.  Tracing statements defined within this file will be
; enabled during TCPIP server initialization.
;
; Note:  For most situations that require tracing, use of the OBEYFILE
; command to dynamically enable and disable tracing may be more
; advantageous than defining trace statements within this file.
;
; NOTRACE disables all TCP/IP tracing.  For detailed information about
; tracing, see the description of the TRACE statement in the
; "IBM TCP/IP for VM Planning and Customization" manual.
; ----------------------------------------------------------------------
NOTRACE ALL

;  TRACE     MOST
;  LESSTRACE ALL
;  MORETRACE MOST
;  SCREEN
;  NOSCREEN
;  FILE TCPIP PROB0001 A

; ----------------------------------------------------------------------
; Control trace file timestamps.
; ----------------------------------------------------------------------
TIMESTAMP 0

; ----------------------------------------------------------------------
; Define user IDs to receive messages when serious errors occur.
; ----------------------------------------------------------------------
INFORM
  OPERATOR
ENDINFORM

; ======================================================================
; Various TCPIP initialization, operation and authorization
; parameters.
; ======================================================================
; Assorted Parameters
; ----------------------------------------------------------------------
; ASSORTEDPARMS
;   NOFWD
;   VMDUMP
;   NOUDPQUEUELIMIT
;   RESTRICTLOWPORTS
;   PERMITTEDUSERSONLY
; ENDASSORTEDPARMS

; ----------------------------------------------------------------------
; Define user IDs that are allowed to issue OBEYFILE and other
; restricted commands.
; ----------------------------------------------------------------------
OBEY
  OPERATOR TCPMAINT MAINT SNMPD SNMPQE ROUTED REXECD DHCPD
ENDOBEY

; ----------------------------------------------------------------------
; Restrict specific user IDs (or a group of similarly named user IDs)
; from using TCP/IP services.
; ----------------------------------------------------------------------
; RESTRICT
;   VEND*
;   BADGUY
; ENDRESTRICT

; ----------------------------------------------------------------------
; Permit specific user IDs to use TCP/IP services.
; ----------------------------------------------------------------------
; PERMIT
;   GOODGUY
; ENDPERMIT

; ----------------------------------------------------------------------
; Keep-Alive Packet parameters
; ----------------------------------------------------------------------
KEEPALIVEOPTIONS
  INTERVAL     20
  SENDGARBAGE  FALSE
ENDKEEPALIVEOPTIONS

; ----------------------------------------------------------------------
; SNMP-related Statements
;
; SYSCONTACT defines the contact person for this managed node, as well
; as how to contact this person.  This definition supplies the value
; for the SNMP MIB variable, "sysContact".
;
; SYSLOCATION defines the physical location of this node.  This
; definition supplies the value for the SNMP MIB variable,
; "sysLocation".
; ----------------------------------------------------------------------
SYSCONTACT
  Main Operator (555-1234)
  P/390 SYS1
ENDSYSCONTACT

SYSLOCATION
  P/390 Operating Theatre
ENDSYSLOCATION

; ======================================================================
; Server Virtual Machine-related Statements
; ======================================================================

; ----------------------------------------------------------------------
; Xautolog the server machines identified via the AUTOLOG statement.
; ----------------------------------------------------------------------
AUTOLOG
; BOOTPD   password    ; BootP Server
; DHCPD    password    ; DHCP Server
  FTPSERVE FTPSERVE    ; FTP Server
; TFTPD    password    ; TFTPD (Trivial FTP) Server
; LPSERVE  password    ; LP Server
; NAMESRV  password    ; Domain Name Server
; NCSGLBD  password    ; NCS GLBD Server
; NCSLLBD  password    ; NCS LLBD Server
; PORTMAP  password    ; Portmap server
; REXECD   password    ; REXEC Server
; ROUTED   password    ; RouteD Server
; SMTP     password    ; SMTP Server
; SNMPD    password    ; SNMP VM Agent Virtual Machine
; SNMPQE   password    ; SNMP VM Client Virtual Machine
; UFTD     password    ; UFT Server
; VMNFS    password    ; NFS Server
ENDAUTOLOG

; ----------------------------------------------------------------------
; Reserve ports for specific server machines.  Port values used are
; those defined in RFC 1060, "Assigned Numbers"
; ----------------------------------------------------------------------
PORT
  20   TCP FTPSERVE  NOAUTOLOG ; FTP Server
  21   TCP FTPSERVE            ; FTP Server
  23   TCP INTCLIEN            ; TELNET Server
; 25   TCP SMTP                ; SMTP Server
; 53   TCP NAMESRV             ; Domain Name Server
; 53   UDP NAMESRV             ; Domain Name Server
; 67   UDP BOOTPD              ; BootP Server
; 67   UDP DHCPD               ; DHCP Server
; 69   UDP TFTPD               ; TFTPD (Trivial FTP) Server
; 111  TCP PORTMAP             ; Portmap Server
; 111  UDP PORTMAP             ; Portmap Server
; 135  UDP NCSLLBD             ; NCS LLBD Server
; 161  UDP SNMPD               ; SNMP Agent
; 162  UDP SNMPQE              ; SNMPQE Agent
; 512  TCP REXECD              ; REXECD Server (REXEC)
; 514  TCP REXECD              ; REXECD Server (RSH)
; 515  TCP LPSERVE             ; LP Server
; 520  UDP ROUTED              ; RouteD Server
; 608  TCP UFTD                ; UFT Server
; 750  TCP VMKERB              ; Kerberos Server
; 750  UDP VMKERB              ; Kerberos Server
; 751  TCP ADMSERV             ; Kerberos Database Server
; 751  UDP ADMSERV             ; Kerberos Database Server
; 2049 UDP VMNFS               ; NFS Server
;
; (End of PORT reservations)

; ======================================================================
; TELNET Server Configuration Statements.
; (The TELNET Server is an "internal client" of the TCPIP server).
; ======================================================================
; INTERNALCLIENTPARMS
;   PORT         23
;   INACTIVE     0
;   TIMEMARK     600
;   CCSTERMNAME  TCPIP
;   CONNECTEXIT  TNEXIT1
; ENDINTERNALCLIENTPARMS

; ======================================================================
; DEVICE and LINK statements
;
; Define the network interfaces used in your environment.
; Hardware definitions
; ======================================================================

; 3172 44A/44B TOKEN-RING
DEVICE UNITY LCS 0700
LINK TR44A ETHERNET 0 UNITY

; ======================================================================
; Routing Statements
;
; Define the routing information required for your environment via the
; HOME, GATEWAY, ARPAGE, BSDROUTINPARMS and PRIMARYINTERFACE statements.
; ======================================================================
;
; ----------------------------------------------------------------------
; Flush the ARP tables every nn minutes (a 5 minute interval is used
; below).
; ----------------------------------------------------------------------
ARPAGE 5

; ----------------------------------------------------------------------
; Primary interface Definition
; ----------------------------------------------------------------------
; PRIMARYINTERFACE  TR44A

; ----------------------------------------------------------------------
; Define the internet (IP) address(es) for this VM host
; ----------------------------------------------------------------------
HOME
;
; local host InterNet addresses  (SYS1)

   192.168.0.234 TR44A

; (End of HOME address information)

; ----------------------------------------------------------------------
; Routing Information
; ----------------------------------------------------------------------
; Note:
;  * Routes defined via the GATEWAY statement are STATIC routes.
;  * Routes defined via the BSDROUTINGPARMS statement are DYNAMIC
;    routes.
; ----------------------------------------------------------------------
; Static Routing Information
; ----------------------------------------------------------------------
GATEWAY
 192.168.0        =     TR44A    1500    0   0
; ----------------------------------------------------------------------
; Define The DEFAULT route used for any network not explicitly routed
; via the previous entries.
; ----------------------------------------------------------------------
;
  DEFAULTNET 192.168.0.2   TR44A     1500 0 0
;
; (End of GATEWAY Static Routing information)


; ----------------------------------------------------------------------
; Dynamic Routing Information
; ----------------------------------------------------------------------
;
; BSDROUTINGPARMS FALSE
;
; ; Link Name  MTU          Metric   Subnet Mask     Destination Address
; ; ---------  -----------  -------  --------------  -------------------
    TR44A      1500         0        255.255.255.0   0
; ;
;
; ENDBSDROUTINGPARMS

; ----------------------------------------------------------------------
; TRANSLATE statements indicate relationships between internet
; addresses and network addresses.  The TRANSLATE statement is most
; often used only in conjunction with HYPERchannel (HCH) and X.25
; network connections.
; ----------------------------------------------------------------------
; TRANSLATE

; (IP) Network    Network   Network Address (see       Link
; Address         Type      documentation for details) Name
; ------------    --------  -------------------------  ---------

; (End of Translate information)
;

; ----------------------------------------------------------------------
; Start all network interface devices used in this environment.
; ----------------------------------------------------------------------
START UNITY

; ======================================================================
; End of PROFILE TCPIP for P/390 (SYS1)
; ======================================================================