Python - Django join on a field that isn not the ForeignKey?

I'm new to Django but I'm making a game where one of the elements is that players vote for each other.

Here are the models I've set up (relevant fields only)

#models.py
class Game(models.Model):
  gamecode = ShortUUIDField(length=4, max_length=4, unique=True)
  phasenumber = models.IntegerField(default=1)
  isActive = models.BooleanField(default=True)

class Player(models.Model):
  game = models.ForeignKey(Game, on_delete=models.CASCADE)
  user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
  isPlaying = models.BooleanField(default=True)

class PlayerVote(models.Model):
  byplayer = models.ForeignKey(Player, on_delete=models.CASCADE)
  forplayer = models.ForeignKey(Player, on_delete=models.CASCADE, related_name="voteforplayer")
  gamephasenumber = models.IntegerField()
  timestamp = models.DateTimeField(auto_now_add=True)

When a user joins a game, they get an entry in the "Player" model and when they cast a vote for another player, an entry is added to the "PlayerVote" model showing which player voted (byplayer) and who they voted for (forplayer).

What I'd like to do is create a QuerySet which contains each player and how many votes they got.

I have tried using annotate which seems to be able to give me a count of the number of votes, but it defaults to joining on the byplayer field, so I end up with the number of votes each player cast, rather than the number of votes each player received.

In SQL I would just do a join on Player.pk = PlayerVote.forplayer.pk, is it possible to do something similar in Django?

Alternatively, I could create an extra model that summarises the votes each player has received and increases by 1 for every vote cast but this seems unnecessary