Currently we check with isinstance, in torch they do this:
https://github.com/pytorch/pytorch/blob/449b1768410104d3ed79d3bcfe4ba1d65c7f22c0/torch/autograd/__init__.py#L448
Which is slightly different, we may want to use the is_tensor_like function from torch to remain compatible.