The mainframe is a computer specialized on batch processing, a conventional PC is used for purpouses such as graphic processing or lightweight computing. Mainframes depend on parallelization and without that they are pretty weak. With batch processing the mainframes uses a software that is something like a job scheduler. The purpouse of this is to allocate cores of the mainframe for performing a certain task in batches. For example a job to sum 1+1 (with a COBOL/REXX program or whatever) would do that task in batch.
When designing an operating system for a mainframe one has to take in account that multicore scheduling is a very important thing to do.
And with all of that, mainframes have devices which are not like your conventional USB mouse, they can be anything and there is no notice of when it can be removed
This guide is intended towards seasoned OS developers who know the theory behind operating systems - this guide assumes the reader already knows what a stream is, why a memory allocator is important for dynamic management and in general the theory behind managing the resources of a machine as an abstraction (aka. theory behind a HAL), etc. If you don't know those concepts the OSDev Wiki is a pretty good place to learn the theory behind operating systems.
Do you want a microkernel that is very resitant to rogue modules - or do you want a monolithic kernel that prioritizes speed over reliability?
It's all up to you. A networking kernel may be an interesting approach. Kernels with a graphical user interface in the mainframe world are rare - if they even exist - but they may be very interesting as well.
You may also want an ultra-portable kernel which can run in any computer including mainframes.
Or perhaps write a kernel with backwards compatibility with MVS/TPF?
You can also try to run an hypervisor to run z/Linux atop of your kernel.
Next