DO NOT MERGE: Bootstrap prototype

[davazp]

This is a pull request just to keep track of obstacles for boostrapping.

I will not merge this as is. I do not plan to make make.sh bootstrap in two stages anymore, but instead to provide

./make.sh --sbcl
and
./make.sh --jscl=<path>

so we can build jscl with sbcl or itself. Similarly for run-tests.sh.

However, I will probably provide some script to bootstrap the full system as a test and part of the CI.

I will open small PRs to apply some of these changes more incrementally and cleanly.

Status

./make.sh is (temporarily) generating out/jscl-xc-node.js as usual with sbcl. After that, in node out/jscl-node.js I can do:

(load "jscl.lisp")
;; this works

(jscl-xc::bootstrap "out/" "test")
;; goes through with a few errors

However when running out/test.js, there is an error when loading toplevel.lisp (the last file!), complaining that *environment* is undefined.

List of changes

JSCL-XC package

Define all jscl into a package jscl-xc. This package exists since (load "jscl.lisp") all the way until the .js file is loaded. And only at the very end, packages are renamed to their final name.

This is for simplicity. I tried reading jscl-xc but changing the target package in dump-symbol but it got tricky.

jscl-target feature

Introduce a feature #+jscl-target. Equivalent to our existing jscl-xc. But I wanted to avoid xc (crosscompile). Because from jscl -> jscl is not technically cross compiling, but bootstraping. Anyway. I can rename it later.

Most of the flags are changed to use #+jscl-target except one in load.lisp I think. Most of src/ should be host independent, so it's natural. We just care about if we are bootstrapping or not. jscl.lisp will care more about the host though, as it needs to run git and do some external things.

Macro readers

Add basic macro reader characters.

This is using still the JS placeholder layer and #j explicit macro reader, even when the host is jscl instead of sbcl. I thought it is nicer to have a single path, and only use CL standard infrastructure from the host.

This is nice because for example, we might want to change FFI. But JSCL should always provide a (close to) standard CL.

with-compilation-unit

Does load also need to delay warnings?

(setf macro-function)

Some defstruct fixes

740644c Fix defstruct predicate
617f51a Fix: defstruct should define a type

Problems

Undefined *environment*.

It's caused by def!struct not creating real classes:

(defun structure-p (obj)
  #+jscl-target (eql (object-type-code obj) :structure)
  #-jscl-target (typep obj 'structure-object))

When bootstraping itself from jscl, the first branch is fine. But the second one will not work.

You can see this if you go to the REPL in master and try this:

(typep (jscl::make-lexenv) 'structure-object)
; => NIL

[kchanqvq]

Thanks for the write up! This clears up a misunderstanding from me: I thought using JSCL instead of other CL to compile JSCL is rather an exercise on conformance than bootstrap (because any conforming CL should be able to compile JSCL). But turns out it is indeed an exercise on bootstrap because we need to handle clashing between host/target JSCL package, etc.

Introduce a feature #+jscl-target. Equivalent to our existing jscl-xc. But I wanted to avoid xc (crosscompile). Because from jscl -> jscl is not technically cross compiling, but bootstraping. Anyway. I can rename it later.

I think xc is fine. I think technically this is usually still cross compiling because we will be compiling from a different (earlier) version of JSCL. Anyway, SBCL use sb-xc.

Macro readers

At some point we need to overhaul the reader, but I repeat myself :) A conforming reader is required to be thoroughly read-table based.

[vlad-km]

Any suggestions for compiling on windows?

[davazp]

Any suggestions for compiling on windows?

What are the main limitations now? the lack of shell? I guess WLS or just doing the build in sbcl directly like

$ sbcl
(load "lisp.js")
(jscl-xc:bootstrap "...") 

Or is there any additional limitations? I imagine paths might be quite broken now. But something we should fix.

Let's think about it when I open a PR to improve the build system to allow specify jscl as a host.

[davazp]

A mismatch in bootstrapping was caused by this

(read-from-string "|toString|") => <#FUNCTION>

Kind of funny! Thank you JS prototype chain. Prelude.js was good but packages created from package.lisp weren't.

Probably good to migrate packages to use hash tables /map too at some point.

[davazp]

Claude was almost completely useless for these bootstrapping issues. Can't grasp the root cause of the issues ever. It always suggests to add #+jscl-target when loading jscl.lisp.

It only helped me to compare jscl-xc.js and jscl.js, looking at the huge diff and categorizing what kind of differences are there.

It makes it more fun though. It's quite cool to hack inside out/jscl-xc-node.js vs sbcl vs dist/jscl-node.js and think of code to trigger differences and examine the state.

[davazp]

Ok! jscl.lisp is compiling itself now 🎉

There were quite a few hacks. Especially defstruct.lisp vs structures.

Many tests are still failing

1824/1873 test(s) passed successfully.

but it's at the point where I build jscl from jscl and I get a pretty close jscl.js and a functioning REPL.

It might be a lot of effort to recreate individual PRs. Maybe it's better to refine this one and fix or create issue to solve all the hacks that were needed.

[kchanqvq]

It might be a lot of effort to recreate individual PRs. Maybe it's better to refine this one and fix or create issue to solve all the hacks that were needed.

Does it make sense to split out the part that implements functionality (i.e. improve conformance, like the time related functions) from the part that make JSCL-from-JSCL work? It looks like there are quite a few such things, which we can merge cleanly to get immediately more functionalities.

I think conceptually the following goals are in order:

1. Build JSCL from any other mostly conforming CL (done).
2. Make JSCL a mostly conforming CL.
3. Build JSCL from JSCL.

Improvements in 2 will probably reduce hacks needed in 3. And it does seem you completed some tasks in 2 in order to make 3 work! I think if we separate them out and consider it under premise of 2, it's easier to review and produce good code. No hacks or magic required in 2!

[davazp]

Let me work on rebasing the PR and try to clean up the commits a bit to try to match that order, and make sure tests (at least the --prebuilt one as default) work.

Some of my fixes can be merged but others are very hacky, def!struct and reader for instance. And I'd prefer if we do them properly.

I'll try to extract the good stuff from it.