Using CC/CSE PCs for Serious Computation

What/Why

• CSE, the CC, and NCL have dozens/hundreds of fancy PCs usually lying idle.
• We’re allowed remote logins and 1 or 3 GB of disk quota, without root access.

Environment TLDR

You get:

• A lot of CPU time/power
• 16 - 64 GB of memory, and swap
• Internet access
• A Linux kernel that services your program’s syscalls

And you don’t get

• Storage
• Root access or apt
• Specific Linux versions (shouldn’t matter)
• Downtime predictability (Use uptime to check if it’s usually up.)
• Up-to-date library or GLibC versions. (Bit me earlier.)

Working Without Root Access

• You might not need to!
• For binaries/libraries, user installations inside the home directory might be enough. You might have to export PATH, LD_LIBRARY_PATH, CC, CXX, CPATH, etc.
• Pro tip: env, env | sort.
• Doesn’t always work out, some packages like gcc actually have dozens of dependencies.

If/When you need Root Access

• Rootless runc
• TODO.

Working around the Disk Quota

• Disk usage (of a file, not of a user): du --summarize --human [optional path].
• Or du -sh [path]
• Home directory: echo $HOME or echo ~ or cd
• Saving big files in /tmp might help a bit.
• Compiling serious software is almost impossible. Try scping precompiled executables, and/or -static linking. Running it should be far easier, because that interfaces mostly with the Linux kernel, and not “build dependencies”.
• Try compiling directly on the project server, and running the output executable on the workstation.
• ldd something.exe tells which libraries it uses. Add those path(s) to LD_LIBRARY_PATH.

SSHFS

• Mount disks/directories from a project server! (Look up man mount.)
• Non-root users cannot exactly mount, but can use sshfs - sshing into a filesystem.
• (Internals) sshfs uses FUSE - Filesystem in UserSpacE to run without root permissions.
• I’m pretty sure not all of this is safe on our unsafe LAN, just safe enough. (Thanks Aman (for finding the vulnerabilities, not for making it unsafe.).)

GUI-Mounting from the File Explorer

• If you’re actually sitting in the lab!
• Ubuntu’s file explorer also allows a GUI mount: Other locations -> Connect to server. (You can try it locally.)
• If sshfs isn’t installed, you can ssh -X into that system, open nautilus, and do your magic. (Try it locally first!)

If Neither Works

• If sshfs isn’t installed (it usually isn’t), then run it inside the container. Welp, good luck.

IP Addresses

• CSEWS (accessible with a CSE LDAP account only!) - Intel Core 172.27.19.[1-60] - i9, 14th gen. 64 GB RAM, 32 GB swap, 24 physical cores, 32 logical cores :).
• CSEWS part 2 - 172.27.19.[160-210] - Intel Core i7, 13th generation
• CSEWS PCs have DNS names - csews33.cse.iitk.ac.in, or simply csews33 if you are on CSE’s LAN.
• Some CC lab - 172.31.4.[100-210] - Unpredictable PCs, try logging in to at least 5.
• Better to use a PC you can physically access if needed. Go to a lab and run hostname -I.
• Help me fill up this list!

pranjal@lab ~/junk % ping ccpc182.cc.iitk.ac.in         
PING ccpc182.cc.iitk.ac.in (172.31.4.182) 56(84) bytes of data.

After Logging in

SSH Examples (for Newbies)

• The command should be about as simple as ssh psingh21@172.31.30.29, or ssh psingh21@ccpc209.cse.iitk.ac.in.
• If it works out, remember, you are now feeding commands to a different computer. Local filesystem paths like /home/ooga/booga won’t work anymore!
• ctrl-C also does not work - use ctrl-D, exit, or sometimes “~.”.
• Check with an LLM if something goes wrong.

Internet Access

wget https://cse.iitk.ac.in/users/prsingh/manuals/authenticator.py
python3 authenticator.py

Network Issues

• ping <url>, ping <IP address> - should take under 5 ms on a LAN connection, under 30 ms on WiFi.
• Disconnect from iitk-sec and use iitk (with or wo authentication).
• If nothing works, ssh into some reachable server and then do a nested ssh to the host.

Utilities to Look for

• tmux
• sshfs
• You’ll need a text editor at some point, try nano and vi (or vimtutor)

Library/Binary Mismatches using strace

• Slightly difficult - strace tells which files were opened or accessed. If there is a mismatch, the strace should end with a failed open sort of call.
• Beautiful, helpful error messages
• strace --follow-forks
• Maybe -e trace=%file
• --failed-only
• Check the manual for other flags.
• For example, these lines tell which copy/version of the library was used: the binary tries all all specified paths till one hits, or exits.

newfstatat(AT_FDCWD, "/data/pranjal/junk/hashtable_run/avx512_1", 0x7ffe4fa511f0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/data/pranjal/junk/hashtable_run/x86_64/libcudart.so.11.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/data/pranjal/junk/hashtable_run/x86_64", 0x7ffe4fa511f0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/data/pranjal/junk/hashtable_run/libcudart.so.11.0", O_RDONLY|O_CLOEXEC) = 3

What/When to Use

• I don’t really know.
• For some workloads (most?) like our single-threaded ultra-slow GPU simulator, Intel Core (the desktop variant) outperforms Intel Xeons (server-grade CPUs)
• Hypothesis - Intel Xeons are optimized for multiprocessing, not single-thread performance.
• We tried Ola Krutrim’s even fancier AMD Epycs, which was twice as slow as our 5yo Intel Xeons for some reason. (And thrice as slow as our 5 yo Intel Core i9.) (“Our” - PROSPAR’s.)
• Probably because that is a shared environment, and the last-level cache (L3) is shared with 63 other cloud clients.

Peeking around

• man lscpu
• lsmem
• Check the number of single-threaded cores
• Check the fastest core’s clock speed
• Are there other users? who, htop, top
• Cleaning up your mess (and logins): pkill -u <username>, or pkill -u $USER

An Example

• 16 physical cores: 0-15
• 8 physical cores support multithreading, and have two logical cores
• Physical cores 0-7 have logical cores (CPUs) 0-15
• Some physical cores are faster than others - 4, 5 support 5200 MHz.

prsingh@csews201:~$ lscpu -e
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ      MHZ
  0    0      0    0 0:0:0:0          yes 5100.0000 800.0000 3669.942
  1    0      0    0 0:0:0:0          yes 5100.0000 800.0000 3505.654
  2    0      0    1 4:4:1:0          yes 5100.0000 800.0000 3665.952
  3    0      0    1 4:4:1:0          yes 5100.0000 800.0000 3701.269
  4    0      0    2 8:8:2:0          yes 5100.0000 800.0000 3699.990
  5    0      0    2 8:8:2:0          yes 5100.0000 800.0000 3700.009
  6    0      0    3 12:12:3:0        yes 5100.0000 800.0000 3700.009
  7    0      0    3 12:12:3:0        yes 5100.0000 800.0000 3699.990
  8    0      0    4 16:16:4:0        yes 5200.0000 800.0000 3683.475
  9    0      0    4 16:16:4:0        yes 5200.0000 800.0000  800.000
 10    0      0    5 20:20:5:0        yes 5200.0000 800.0000 3674.728
 11    0      0    5 20:20:5:0        yes 5200.0000 800.0000  800.000
 12    0      0    6 24:24:6:0        yes 5100.0000 800.0000 3700.000
 13    0      0    6 24:24:6:0        yes 5100.0000 800.0000 3675.292
 14    0      0    7 28:28:7:0        yes 5100.0000 800.0000  799.274
 15    0      0    7 28:28:7:0        yes 5100.0000 800.0000  800.000
 16    0      0    8 32:32:8:0        yes 4100.0000 800.0000 3099.999
 17    0      0    9 33:33:8:0        yes 4100.0000 800.0000 3100.014
 18    0      0   10 34:34:8:0        yes 4100.0000 800.0000 3099.999
 19    0      0   11 35:35:8:0        yes 4100.0000 800.0000 3100.024
 20    0      0   12 36:36:9:0        yes 4100.0000 800.0000 3100.005
 21    0      0   13 37:37:9:0        yes 4100.0000 800.0000 3100.023
 22    0      0   14 38:38:9:0        yes 4100.0000 800.0000 3099.990
 23    0      0   15 39:39:9:0        yes 4100.0000 800.0000 3099.992

• CSEWS47 is a 14th generation CPU - behold 5800 MHz.

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ   MINMHZ      MHZ
  0    0      0    0 0:0:0:0          yes 5500.0000 800.0000 3438.629
  1    0      0    0 0:0:0:0          yes 5500.0000 800.0000  800.000
  2    0      0    1 4:4:1:0          yes 5500.0000 800.0000 5500.000
  3    0      0    1 4:4:1:0          yes 5500.0000 800.0000  800.000
  4    0      0    2 8:8:2:0          yes 5800.0000 800.0000  800.704
  5    0      0    2 8:8:2:0          yes 5800.0000 800.0000  800.000
  6    0      0    3 12:12:3:0        yes 5800.0000 800.0000 5307.895
  7    0      0    3 12:12:3:0        yes 5800.0000 800.0000  800.000
  8    0      0    4 16:16:4:0        yes 5500.0000 800.0000  799.153
  9    0      0    4 16:16:4:0        yes 5500.0000 800.0000  800.000
 10    0      0    5 20:20:5:0        yes 5500.0000 800.0000  799.650
 11    0      0    5 20:20:5:0        yes 5500.0000 800.0000  800.000
 12    0      0    6 24:24:6:0        yes 5500.0000 800.0000 3757.607
 13    0      0    6 24:24:6:0        yes 5500.0000 800.0000  800.000
 14    0      0    7 28:28:7:0        yes 5500.0000 800.0000  800.000
 15    0      0    7 28:28:7:0        yes 5500.0000 800.0000  800.000
 16    0      0    8 32:32:8:0        yes 4300.0000 800.0000  800.000
 17    0      0    9 33:33:8:0        yes 4300.0000 800.0000 4305.543
 18    0      0   10 34:34:8:0        yes 4300.0000 800.0000  800.000
 19    0      0   11 35:35:8:0        yes 4300.0000 800.0000  800.000
 20    0      0   12 36:36:9:0        yes 4300.0000 800.0000  800.000
 21    0      0   13 37:37:9:0        yes 4300.0000 800.0000 4261.242
 22    0      0   14 38:38:9:0        yes 4300.0000 800.0000  800.000
 23    0      0   15 39:39:9:0        yes 4300.0000 800.0000  800.000
 24    0      0   16 40:40:10:0       yes 4300.0000 800.0000  800.000
 25    0      0   17 41:41:10:0       yes 4300.0000 800.0000  800.000
 26    0      0   18 42:42:10:0       yes 4300.0000 800.0000  800.000
 27    0      0   19 43:43:10:0       yes 4300.0000 800.0000  800.000
 28    0      0   20 44:44:11:0       yes 4300.0000 800.0000  800.000
 29    0      0   21 45:45:11:0       yes 4300.0000 800.0000  800.000
 30    0      0   22 46:46:11:0       yes 4300.0000 800.0000  800.000
 31    0      0   23 47:47:11:0       yes 4300.0000 800.0000  800.000

Credits

• The beautiful AWS-proof IITK LAN.
• Shiv told me about rootless runc.
• Converted from markdown to HTML using pandoc.
• People who pointed out mistakes/bugs: zero so far.