Proposal - Let's create an accurate source of truth

[luismoyano]

Hello Json Logic!

While working on shiny json logic I've been observing a few inconsistencies in some tests which I shared here and also found that the truthiness of the tests on the hub is not correctly updated with the latest desicions.

Given this I believe json logic org is falling into a case of truth scattering as it's hard to determine which set of tests validates the standard. Is it the existing in compat-tables, the set within json-logic-engine or is it the set existing in this hub?

In the @jwadhams era we were fully able to understand what was the standard made of with the official list of tests in jsonlogic.com and could work on different lang implementations, right now as a maintainer it is hard for me to check where should I be standing...

I'd gladly take on the work of aggregating the valid tests in the same place so everyone can have access to them, but need your input to define which ones make it and where.

Thanks!!

[TotalTechGeek]

Hey! One of the very first things we did was establish voting procedures and a way for tests to become accepted by the community.

These are the accepted proposals and their phases:

• https://github.com/json-logic/.github/blob/main/ACCEPTED_PROPOSALS.md

And these are all of the tests that have been approved all in one folder:

• https://github.com/json-logic/.github/tree/main/tests

The hub is the source of truth. The proposal process drives which tests go in there.

The compat-tables repo includes some things that hasn't gone through the proposal process, it was to rapidly prototype the idea. We were planning on firming some things up over there with a bit more adoption on the proposals :)

[TotalTechGeek]

and also found that the truthiness of the tests on the hub is not correctly updated with the latest desicions.

Er -- I've double checked, the tests (rule / data) in the hub do reflect the most recent decisions.

[luismoyano]

@TotalTechGeek Thank you so much for double checking! The reason for which I say there is starting to develop a case of truth scattering around the org is because it is currently hard to understand which on should be the source, let me explain:

We can intuitively understand that the source of truth is hosted in the hub as you stated above, but the set of tests in compat's table is different both by number and quality, there's a discrepancy with truthiness here and here as an example where we can see the definitions of truthiness are the opposite from one to the other.

Then again compat-tables includes three tests that seems to be incompatible with Wadhams' implementation as I described here Where we can see how Wadhams' original tests.json includes a case where it returns an empty array on map operations whenever we happen to find the initial collection to be missing within the operator while compat expects an error to be thrown even though one of the pillars of this org is retrocompatibility as stated in the accepted proposals.

So as a maintainer who wants to keep its implementation as close to the standard as possible, the first place to look at should be the compat tables as they visually allow anyone to understand how close or how far one is from achieving the standard, But the three sets of test cases involved here (Wadhams', Hub's and compat tables') include discrepancies between each other!

I don't think this is a problem that fixes itself with a couple of PR's to a repo, otherwise I'd be glad to do it myself, but to my eye this will keep happening as long as we keep having separate databases for functionalities that should all feed from only one - a.k.a The source of truth.

Because of this I wanted to bring this issue I found to the community's attention in order to decide on actions and tackle the issue.

Hope this clarifies my findings better. It seems to me that this has been happening because there's some part within the process that needs to be improved.

Again thanks for your time and dedication to JSON Logic, I've been so excited to know about these efforts since I got in touch!

[luismoyano]

Sorry for the second comment, but while working on achieving the standard I found very confusing to have the preserve keyword already in compat tables but not a single mention of it in the accepted proposals, then I found out it is still in the voting phase and not yet approved which seems to me another race condition that occurred in the process, thus could still be rejected or open to changes, but when trying to measure compatibility, it is already having an impact against all the tested libraries.

[luismoyano]

@codetiger After getting a hold of @TotalTechGeek and having a conversation I got to understand that we should be using the test suite at .github as the source of truth. Given that there are some substantial differences between that test suite and the compat-tables' one. What do you think of pulling the tests cases from the .github repo instead of having a proprietary test suite?

[codetiger]

yes, the test suite in compat-tables repo seems to be an old copy of the source of truth. It does not seem to have the latest decision related changes. We can make it use the source of truth link. @TotalTechGeek and other members please share your thoughts on this.

[luismoyano]

@gregsdennis, @toddbaert Can we have a few words of opinion here? ☝️