Support covariant typeclass families

[nigredo-tori]

This issue is motivated by the Caliban library, which defines a Schema typeclass like this:

abstract class Schema[-R, T]

R can be different for different T, but we can combine heterogenous instances. For example, for some case class with fields that have corresponding instances Schema[R1, Field1] and Schema[R2, Field2] we can build a Schema[R1 & R2, SomeCaseClass]. However, as it stands right now, we can't easily derive it using Magnolia, because we have Schema.Typeclass defined as a Schema[Any, *], so that is what gen produces.

The way Caliban "solves" this is by providing a GenericSchema helper to the library user, with its own specialized Typeclass. It gets awkward to use without giving up the way ZIO code tends to be structured. I won't get into details, suffice to say that I'd like to only have to rely on Schema.gen for derivation.

I imagine that everywhere where we use Typeclass (e.g. here that is what typeConstructor is bound to), we can instead use a narrower type. It's a bit awkward to explain, so I'll add a PR for Scala 2 shortly.

Does this change fit this library, or is it specifically geared towards invariant typeclasses? I feel like with the rise of ZIO we are likely to see more of covariance everywhere, so cases like this could get somewhat more common.

[nigredo-tori]

Hm. It's kind of difficult to do this for Scala 3, since join and split signatures are set in stone, so I don't see how the calling code could define something like this. It may be possible to make them inline...

Since Caliban doesn't need its GenericSchema workaround in Scala 3, I took a gander at their code, and they just roll their own derivation. 🤦‍♂️ They use some minor parts of Magnolia, but not the split/join/derived mechanism or associated classes.

[nigredo-tori]

I don't see a way to implement this on the scala3 branch without breaking backward compatibility. The issue is that on that branch split and join are defined in a trait, which means that the implementors can not introduce new type parameters to steer typing. And whatever else I try, it breaks upon the fact that I can't get a narrower type of the parameter from the calling code:

final case class CaseClass[+Typeclass[_], Type]():
  def build: Typeclass[Type] = sys.error("")

trait Derivation:
  type Typeclass[_]
  // Changed the signature to check more ideas
  inline def join[A](
    inline caseClass: CaseClass[Typeclass, A]
  ): Typeclass[A]

final case class CollectFields[+Out, A]()

object WithOverrideParams extends Derivation:
  type Typeclass[a] = CollectFields[Any, a]
  // Doesn't match `Derivation.join` signature
  override def join[Out, A](
    caseClass: CaseClass[CollectFields[Out, *], A]
  ): CollectFields[Out, A] =
    caseClass.build

object WithMatchReturnType extends Derivation:
  type Typeclass[a] = CollectFields[Any, a]
  type Extract[T] = T match {
    case CaseClass[CollectFields[out, *], a] => CollectFields[out, a]
  }

  override def join[A](
    caseClass: CaseClass[CollectFields[Any, *], A]
  ): Extract[caseClass.type] =
    caseClass.build

  val result: CollectFields[String, String] =
    // Produces `CollectFields[Any, String]` instead.
    join(CaseClass[CollectFields[String, *], String]())

object WithTransparentInline extends Derivation:
  type Typeclass[a] = CollectFields[Any, a]

  override transparent inline def join[A](
    inline caseClass: CaseClass[Typeclass, A]
  ): Typeclass[A] =
    joinImpl(caseClass)

  private def joinImpl[Out, A](
    caseClass: CaseClass[CollectFields[Out, *], A]
  ): CollectFields[Out, A] =
    caseClass.build

  val result: CollectFields[String, String] =
    // Produces `CollectFields[Any, String]` instead.
    join(CaseClass[CollectFields[String, *], String]())

FWICT, the same should be true for scala3-v2. Maybe someone more familiar with Scala 3 can see some trick to achieve this?

I guess I'll try changing the join and split signatures next to give the implementors enough freedom to narrow the result types.

[nigredo-tori]

Oh, also it seems that narrowing doesn't work for recursive data structures. For those we build a recursive lazy val here (I think) for which the compiler requires us to provide a type.

Now this one would be difficult to untangle.

[nigredo-tori]

I've played with some ideas for narrowing generated typeclasses for (mutually) recursive types. It's a mess to do so automatically. The best idea I have in that regard is to eliminate recursion via some fixed-point combinator-like trick, but this will be hard to get right, and the result would probably be way too complicated and fragile.

I think the best way here is to allow the calling code to specify the narrowed typeclass:

def genNarrow[Tc[_], T](
  c: whitebox.Context
)(implicit
  Tc: c.WeakTypeTag[Tc[_]],
  T: c.WeakTypeTag[T]
): c.Tree

I'm going forward with this approach, although it seems this would require a deeper understanding of the implementation than I have now.