There are things in the IT industry whose even existence has become a beautiful myth.

Cray

The knowledge described in this article is extremely rare, as it has previously been held by individuals with an academic degree, special training, and, most importantly, access to the necessary equipment.

There are maybe a hundred engineers left in the world who had a chance to touch the legend.

And only a few of them can still explain something.

An ancient monster from the past was returned to life.. by fans. Who gave ordinary people a chance to explore the legend.

Seymour Cray with one of his supercomputers.

The legend of Cray

Seymour Cray will always be remembered in computer history, as the machines he built were fastest on the planet for decades. Created entirely by hand and at the cost of millions of dollars, Cray supercomputers were built mostly for research labs and large data centers.

And for intelligence agencies around the world:

Cray supercomputers have always been associated with secret projects due to their use in cracking secret codes, passwords, and ciphers.

Interesting fact that Seymour Cray began his career in the US Navy and engaged in breaking Japanese codes during World War II. Seems that he continued his relations with CIA since then.

And you, dear reader, have only one chance to see a Cray supercomputer — in movies, where they appeared since 70s:

Ordinary people shouldn't even dream of working with such machines, even if they were born in the USA.

Permits, specialized training, certification, and typically a PhD were required for "users" of such equipment.

Yes, Cray supercomputers were imported to Russia, but with very high barriers and only simple models. In particular, Roshydromet had a long practice of using Cray machines since 90s.

I myself, despite years in the software development industry, have only heard about them, and seen pictures on the Internet, like this one:

This makes the story told below even more surprising.

Cray Simulator

Andras Tantos' Cray simulator story deserves a Hollywood film adaptation because it represents the engineering triumph over technical difficulties:

So it’s settled. I’m building a Cray-1.

I highly recommend reading this entire story, as it close to attempts of decipher German codes during World War II.

For example, just to read data from discovered tape, Andras had to implement a special driver for the virtual controller, relying just on images like that:

Then it was required to manually restore the boot sequence:

Then dealing with bootloader bugs and network emulation — do not forget that we are talking about a real supercomputer, where all major parts are connected via network.

Andras did absolutely incredible work, without much hope and belief in success.

That's why results are so impressive.

The front panel of the Cray supercomputer with boot indicators. The power button is on top.

Return of the legend

Cray simulator rely on some external applications to operate; those must be installed before the first run:

xterm, telnet, tmux

Also I used cool-retro-term to make more epic screenshots.

Important note:

Telnet clients launched at startup actually connect to the simulator ports, through which console commands are transmitted to the virtual UNICOS and back. This is not a network connection to UNICOS itself.

Real network connection from host to UNICOS requires one more configuration step on the host, and since the installation takes place over the network, this step is required.

So to setup networking, run these commands:

brctl addbr craybr
ip tuntap add mode tap tap1
ifconfig tap1 up
brctl addif craybr tap1
ifconfig craybr 172.16.0.1 netmask 255.255.255.0

Despite its 'visual complexity', there is no anything complicated:

We add a new bridge, named craybr, then create a virtual network interface tap1 and assign IP address 172.16.0.1 to it.

As a final step, this interface is added to our newly created bridge. That's all.

The device name of the network interface is specified in unicos.cfg:

..
EthInterfaces {
			en0 {
				InterfaceNameLinux          tap1                ; Tap interface name to connect to on Linux
				InterfaceNameWindows        "Cray Ethernet"     ; WinTap interface name to connect to on Windows
				SimMacAddr                  0x020143524159
				Channel                     020
				IopNumber                   0
			}
		}
..

But the IP-address must be 172.16.0.1, since the internal interface in UNICOS is hardcoded as 172.16.0.2. And changing this is bit problematic.

We can go further and enable external routing:

brctl setfd craybr 0
sysctl net.ipv4.ip_forward=1
iptables --table nat -A POSTROUTING --out-interface wlan0 -j MASQUERADE

For Mageia Linux, the external interface would be called something like wlp4s0.

Now, from UNICOS execute:

route add default 172.16.0.1

You just released a Cray supercomputer to the Internet! Congrats!

Even if it's virtual:

Everything you see in the console above is original software from Cray, for Cray supercomputers.

Binary build

There are builds of the Cray simulator for 64-bit Linux, with UNICOS versions 10.0.0.2 and 10.0.1.2 installed, created by well-known neozeed.

But these builds are outdated (from 2022), so no guarantees that they would work on your Linux system.

To start Cray simulator from these builds:

./unicos

Don't forget to configure the network before starting (see above), otherwise it will not work.

The UNICOS 10.0.1.2 running in emulation of Cray J90 supercomputer:

Need to notice that I replaced xterm that is used by default with more cool cool-retro-term, for epic screenshots.

For sure, real Cray J-90 had no such ancient terminals and looked much more futuristic:

A slightly newer Cray T-90 with an operator workstation (OWS)

If you look closely at the screenshot upper, can find an SGI workstation with Irix and 4dwm window manager.

That's because Cray supercomputers were operated from workstations, not produced by Cray company.

There were workstations from Sun Microsystems, SGI and even Apple:

So, to use cool-retro-term instead of xterm, change this setting in unicos.cfg:

This is what a fully loaded Cray simulator with running UNICOS looks like:

According to the author, this pre-installed UNICOS has the following accounts:

The root password is 'password' and I've created a neozeed user with the password of 'password' so you can telnet in

Log in as root:

If you have configured everything correctly, would be possible to establish a network connection between the simulator and the host in both directions.

And will be possible to log in over the network, directly to the Cray machine:

To stop simulation, type exit in the bottom console of the main application and press Enter:

The bottom block is also terminal, and allows input.

This version of Cray simulator would be sufficient for the next steps.

If it works on your system, and you don’t have a persistent desire to “compile everything from source,” you can skip the next step and go straight to the stage of truly sophisticated adventures.

The first Cray supercomputers were covered in the genuine leather of dead engineers. — The first Cray supercomputers were covered in the genuine leather ~~of dead engineers~~.

Build from source

This step may not be suitable for everyone and your favorite ChatGPT is unlikely to give advice on such project, but it’s still worth doing for greater immersion.

Cray simulator is written in C++ using the Boost library, so compilation from the source code is quite slow.

Fetch sources:

git clone https://github.com/andrastantos/cray-sim.git

There are no release branches in this repository and everything is compiled from themaster. But before starting, you need to install the following dependencies, for Ubuntu:

 g++ make libboost-all-dev libncurses-dev libgpm-dev

for Mageia Linux:

gcc-c++ make lib64boost-devel lib64ncurses-devel lib64gpm-devel

Simulator sources are located in the simulator directory, so the build process should be launched from there, not from the root folder.

Because there is no static version of the Boost library in Mageia, I used dynamic linking there:

make LINK_TYPE=dynamic build

For Ubuntu this step is not required and build would work by default:

make build

Compiled binaries could be found in the simulator/_bin folder, but the simulator's scripts know about this, so there is no need to copy binaries manually to make it work.

Completing the installation on one of the survived Cray Y-MPs, the console is a real Wyse terminal from that era..

Installing UNICOS

The most interesting stage, which you would miss, if you stop on precompiled build, created by neozeed:

Installing the UNICOS operating system on a Cray supercomputer simulator from original CD images.

Once upon a time ~~in far far galaxy~~, a team of highly qualified and certified engineers was responsible for the deployment process of a supercomputer.

And they carefully guarded the secrets of this process.

But thanks to curious fans, now you can finally take part in it too.

As I mentioned before, these two accidentally discovered disks are the only survived media with the UNICOS boot image. Without them, the story of Cray Simulator would have been completely different.

UNICOS images are available at the following links

Both images are bootable, the first one contains UNICOS 10.0.0.2 for Cray J90, the second — UNICOS 10.0.1.2 for Cray SV1. All installation steps are the same, but I will be installing 10.0.1.2 as “more recent”, from the second ISO image.

Actually, different versions of UNICOS were designed for different supercomputers, but in the simulator, they both work fine.

The original installation steps from the author of Cray Simulator could be found here, but unfortunately they bit outdated, so you'll have to use them, but with minor modifications.

Need to warn that all commands below, executed from the host (and simulator's control scripts), require bash.

On the UNICOS side, ksh is used, but for the installation this is not much important.

At first, we need to set these environment variables:

export ISO=/mnt/iso
export SIM=/opt/work/cray/cray-sim

First variable would contain full path to mounted ISO image with UNICOS, the second — full path to Cray simulator’s root folder.

Mount UNICOS image:

fuseiso ./cray-cd2.iso $ISO

The UNICOS image contains long file names; also, there are files and directories with names in different cases:

So there is no option to use well-known mount -o loop command. Copy the archive with the system and kernel:

cp $ISO/unicos_exe/ram_fs $SIM
cp $ISO/unicos_exe/krn_11.bgz $SIM

Unpack the archive with kernel:

cd $SIM
gzip -d -- < krn_11.bgz | cpio -i

There would be multiple new files in the root directory of Cray simulator; the one we need is called unicos.generic – UNICOS kernel.

We can finally run the simulator, but using the RAMFS image, which we just copied from the installation disk:

./unicos_ramfs

The simulator will start and a terminal will appear with a connection to UNICOS, but running in single user mode:

An operating system for Cray supercomputers. In "Single User Mode".

We need to configure the network on the UNICOS side, since the next step is about copying system files from the mounted ISO image.

Reminding that the connection via telnet actually occurs to the simulator itself, not to the emulated OS inside.

Enter in UNICOS terminal:

ifconfig en0 172.16.0.2

After this, ping to host should work:

Then another interesting stage begins, ~~full of pain and suffering~~, since you have to deal with one very ancient technology for transferring files between computers — rcp.

The UNICOS, which we are trying to launch, is a system from 1997 and there is simply no other tools for transferring files in its boot image.

The entire process of deploying a supercomputer was top secret, so no one bothered about “usability”. One more important note:

The only available tool for transferring files over a network is a client, not a server

In order to connect and download files from host, you need to set up a server — an ancient rcp server on a modern Linux. Due to its age and serious security flaws, rcp in any form (client and server) is not present by default in any Linux or BSD, and its installation and launch in a modern environment requires some “special magic”.

For Ubuntu you need to install these packages:

rsh-client rsh-server

for Mageia Linux:

rsh rsh-server

In Mageia, the rsh server is launched via the xinetd daemon, which is disabled by default and attempting to launch it will raise this error:

xinetd.service is not active, cannot reload.

So you'll need to start xinetd first:

service xinetd start
service rsh start

On the next step, you need to allow the rsh daemon to accept incoming connections

For Ubuntu, you need to add this line to the file /etc/hosts.equiv:

172.16.0.2 +

For Mageia, the same line should be placed in ~/.rhosts

That will allow copying the installation script from the host system to running UNICOS.

But before this, the script needs to be modified.

Open the install file (located in the root directory of Cray Simulator) with your favorite vi editor and fill these variables:

LOCAL_LOGIN = username on host
ISO_MNT = full/path/to/folder/with/mounted/UNICOS/image

Then, we need to set two additional variables:

SIM_LOC = full path to the Cray Simulator's root directory.
UNICOS_EXE = UNICOS_exe

Second variable is required due to a small bug in the install script; the value should be an exact string UNICOS_exe.

In result, there should be something like this:

The final script for installing the Cray operating system, your mom will be proud ;)

Next, from UNICOS (don't forget to replace username and path):

cd /
rcp alex@172.16.0.1:/opt/work/cray/cray_sim/install .

The install file will be copied from the host and will appear in the root of the UNICOS file system:

Run the UNICOS installation script:

./install

The first message with main installation parameters:

Press any key and the exciting process of installing real operating system for the Cray supercomputer will begin:

It's just copying files over the network, but it's epic!

The process takes several hours, regardless of the power of your equipment, so you'll have enough time to ~~make a bloody sacrifice to the dark gods~~ drink tea in good company.

When done, you will be prompted to set a password for the superuser, and a dialog for creating a standard user account will be launched.

And the installation of UNICOS will finally be completed:

Yes, you just installed an OS on a Cray supercomputer, a virtual one.

There is one important step, which occurs during UNICOS installation — kernel liking.

This practice dates back to the early days of UNIX, when there were many architectures and few standards. Hardware compatibility was poor, so kernel linking was used as a 'final test' of the installed system.

OpenBSD, for example, kept this practice, although for a different reason.

Stop simulation by typing exit command and make sure that the main UNICOS kernel has been successfully linked — the file unicos.ymp.10012 should appear:

After this, check the unicos.cfg file, where the link to a new kernel should appear:

If everything is fine and link is in place, launch full simulation:

./unicos

This is what a fully running Cray J90 supercomputer simulator looks like:

If you specify a default route on the UNICOS side, using same command as described above:

route add default 172.16.0.2

..we'll get internet access.

Right from the Cray supercomputer, yep:

Mom, I brought the Cray supercomputer online!

You were hoping I'd settle down, pour some champagne, ~~call girls~~, and go on a bender, weren't you? Of course not, and there is still a lot more interesting stuff.

UI and Cray supercomputer

The UNICOS image we just used for installation contains the working client library for the X11 protocol. Of course, there is no X server inside, because our smart ancestors used dedicated workstations to control Cray supercomputers, like this one from SGI:

However, it's still possible to 'forward' an X11 application, so it will be rendered on the running Xorg server via network:

A clock ticking right on a Cray supercomputer.

These two applications in the screenshot above, xterm and xlock, were launched from a running UNICOS and displayed in the Xorg server on Mageia Linux.

To repeat, you must make a blood sacrifice... uh, just follow these three simple steps described below.

Start Xorg with networking support

By default, even in the most old-school Linux distros like Slackware, the X server starts with the -nolisten parameter, which prohibits remote connections.

To verify, simply run this on host, which will show the running X server process with all parameters:

ps -ax |grep X

The X server is typically launched from a special application called «display manager» (dm), which is responsible for a beautiful graphical authorization window — the login prompt.

So parameters we need to alter are stored in the settings of display manager.

Since my system uses LightDM, I added the following setting to the [Seat:*] section of the file /etc/lightdm/lightdm.conf.d/49-mageia.conf:

xserver-allow-tcp=true
xserver-command=X -listen tcp

Then restart lightdm service:

service lightdm restart

For sure, you will be logged out immediately, so be prepared and stop the Cray simulator, if it is running.

Allow remote access without authentication

The next step is to disable authorization when connecting to the X server over the network.

To do this, log in using DM and launch the graphical environment as you usually do, then enter in the console:

xhost +

It looks like this:

After running this command, everyone will be able to connect remotely to your X server from any host and without authorization. This, of course, considered dangerous, but twenty years ago.

Nowadays, with Wayland instead of Xorg, only a few people even remember about the possibility of such a connection, so don't bother:

anyone who could theoretically connect to your machine in this way has died or enjoying their insanity.

Except the author, of course.

Setup remote Xorg on UNICOS

The last step is to specify the address of the remote X server on the UNICOS side. This is done with this command (don't forget about ksh by default):

setenv DISPLAY 172.16.0.1:0.0

The X11 software is located in the /usr/bin/X11 directory; this is what running xterm looks like, for example:

"This is a private computer facility" - the most exciting invitation in the world.

If you have followed all the steps correctly, a graphical window will appear with the application running in a supercomputer environment:

And... no, this is not the end.

Special toys, for special people

The simulator comes with an interesting archive goodies.tar, compiled by the original author of the simulator, which can be found in the unicos_tools directory.

This archive contains several well-known utilities compiled for UNICOS, like bash and «midnight commander».

Behold, mortals, for this is what your favorite mc looks like running on a Cray supercomputer:x

This wonderful archive is copied using rcp, already known from the installation process, described upper:

rcp alex@172.16.0.1:/opt/work/cray/cray-sim/unicos_toos/goodies.tar /

Unpack into the /usr directory, which is already in the PATH environment variable:

cd /usr
tar xvf /goodies.tar
rm /goodies.tar

Before starting, you need to set the execution bit:

chmod +x /usr/bin/mc
chmod +x /usr/bin/bash

and correct permissions:

chmod 765 /usr/bin/mc
chmod 765 /usr/bin/bash

If mc would complain on start, you need to set the terminal type:

setenv TERM xterm

And no, that's not the end of it either..

(heavy object strikes in the background)

Mechanical voiceover:

..the threat of corporate secrets being exposed has been eliminated.

Proceeding to clear the area.

Russian version.

Сray: Resurrection