[ExI] LLMs and Computability

[Darin Sunley]

Samantha,

I suppose it depends on what you mean by "implement".

Consider the difference between

F(n)=1√5[(1+√52)n−(1−√52)n]

And

def F(n)
  if n < 2
    return n
  else
    return F(n - 1) + F(n - 2)
  end
end

Both implement the exact same function - finding the nth Fibonnaci number,
but using radically different code. The first is implementation of the
second. It happens to be a lossless compression of the second. But what if
it was an imperfect implementation, delivering occasionally different
results from the second. I think it would still be fair to call it an
implementation, albeit an imperfect one.

Similarly, the F(tokenstream -> tokenstream) learned by a LLM is an
implementation, albeit imperfect, of the system that generated its training
data. And if that underlying system (human linguistic computation) is
Turing complete (modulo memory size), then thel earned function may be as
well.

[If you want to reserve the definition of "implementation" for functions
that are precisely correct over their entire range, I won't argue, but I
think "implementation" is a useful concept if it can encompass some
fuzziness.]]

On Thu, Apr 20, 2023 at 2:17 AM sjatkins via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

>
>
> ------- Original Message -------
> On Wednesday, April 19th, 2023 at 3:50 PM, Darin Sunley via extropy-chat <
> extropy-chat at lists.extropy.org> wrote:
>
>
>
>
>
> That seems like it's the secret sauce - the ginormous hint that got
> ignored by data/statisticis-centric ML researchers. When you learn how to
> map token streams with significant internal structure, the function your
> neural net is being trained to approximate will inevitably come to
> implement at least some of the processing that generated your token
> streams.
>
>
> I think this is too strong a claim.  The LLM system will simulate similar
> features as those found in what generated the token systems but I would
> argue this in not an implementation of those features.  If you will it is
> more like a ghost image or echo of those features.  The mapping is very
> much not the territory.
>
> - samantha
>
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20230420/146b57cd/attachment-0001.htm>