what you probably want to know about disabling SSH keys provided by gpg-agent is in this... take note of the ! syntax:
[grahamc@Petunia:~]$ cat .gnupg/sshcontrol
# List of allowed ssh keys. Only keys present in this file are used
# in the SSH protocol. The ssh-add tool may add new entries to this
# file to enable them; you may also add them manually. Comment
# lines, like this one, as well as empty lines are ignored. Lines do
# have a certain length limit but this is not serious limitation as
# the format of the entries is fixed and checked by gpg-agent. A
# non-comment line starts with optional white spaces, followed by the
# keygrip of the key given as 40 hex digits, optionally followed by a
# caching TTL in seconds, and another optional field for arbitrary
# flags. Prepend the keygrip with an '!' mark to disable it.
# RSA key added on: 2017-10-07 10:02:25
# Fingerprints: MD5:96:1d:d7:e3:28:5d:e6:9b:c0:34:b0:1a:01:4c:c9:a9
# SHA256:bgEmSPrLeuDOcM8QgjgzIfVlyRo1IeBTT9z1I8sAR8I
3B493449D68C22E92DB9F9048E1E32A702364175 0
# RSA key added on: 2018-02-05 13:48:13
# Fingerprints: MD5:f8:72:96:e7:ef:e4:a3:46:39:d4:86:bb:8e:1b:1b:4b
# SHA256:9lJwrnDo0MI/osV7Eusz2kPWMVney09NFypkBFrzm28
DB1A6B6A24627F9D7551B21A9D51AB2C0376A4B2 0
# RSA key added on: 2018-03-04 21:28:44
# Fingerprints: MD5:3d:6d:a9:ac:76:e7:48:ad:4a:d0:50:81:46:2f:25:94
# SHA256:M2xGJvDL8hSCbe0GE6qDLBL7AOFPjq6C9iME6wRaZ58
A6FEF12F918389D277ACD84CF8B268835FA1D24A 0
# RSA key added on: 2018-05-09 19:15:48
# Fingerprints: MD5:ee:f1:24:85:d1:e2:a5:29:35:82:72:d0:02:f3:e3:14
# SHA256:TiIUAw/w6kByxDrWBXi5pszUrkMvXWwtGX5TVkzELSk
B156709A4B3C39A6BC56E3DE74C194EEEA903CB6 0
# Ed25519 key added on: 2019-01-22 11:21:42
# Fingerprints: MD5:6d:70:cd:3c:ed:4e:1e:d7:30:d9:44:6d:5c:e0:2e:92
# SHA256:nsXG8wmjLBU6KlGob8vw9nwKgyb/kSWZAZbGVRnQCQM
78973E39B5295662BAE63D3A3F8282B2BFABE4DA 0
# ECDSA key added on: 2019-02-25 10:54:24
# Fingerprints: MD5:14:bd:7a:00:96:a4:14:8e:e3:7b:c6:7f:0d:29:c0:ff
# SHA256:YMXioiuSN9I3EWMPAzZ4oHtgQCiygCOndHOs3Kn5xdk
52CF5EF6C4C5BAFC648D92F2DBF724FA887F6811 0
but if you want to know more ...
your SSH keys are stored by gpg-agent, next to your actual GPG secret keys, in:
[grahamc@Petunia:~]$ ls -la ~/.gnupg/private-keys-v1.d
total 62
drwx------ 2 grahamc users 12 Feb 25 2019 .
drwx------ 4 grahamc users 22 Feb 26 13:29 ..
-rw------- 1 grahamc users 1197 Jan 28 2019 3B493449D68C22E92DB9F9048E1E32A702364175.key
-rw------- 1 grahamc users 381 Feb 25 2019 52CF5EF6C4C5BAFC648D92F2DBF724FA887F6811.key
-rw------- 1 grahamc users 2055 Jan 28 2019 620D23E03A7B1FFD95BA9F14A36F861C759B61C1.key
-rw------- 1 grahamc users 359 Jan 28 2019 78973E39B5295662BAE63D3A3F8282B2BFABE4DA.key
-rw------- 1 grahamc users 2071 Jan 28 2019 8EB7C466CD657693847D604D9DEE8A07450DD698.key
-rw------- 1 grahamc users 1215 Jan 28 2019 A6FEF12F918389D277ACD84CF8B268835FA1D24A.key
-rw------- 1 grahamc users 1200 Jan 28 2019 B156709A4B3C39A6BC56E3DE74C194EEEA903CB6.key
-rw------- 1 grahamc users 615 Jan 28 2019 CBB37AFD90A2B7914BB90D2EEFEE6484520DE188.key
-rw------- 1 grahamc users 1018 Jan 28 2019 DB1A6B6A24627F9D7551B21A9D51AB2C0376A4B2.key
-rw------- 1 grahamc users 2055 Jan 28 2019 EAC1248DF7C760480853966B11DF4CF556AAA988.key
good luck with that, but you can see which of those are your GPG keys with they "Keygrip" data in:
[grahamc@Petunia:~]$ gpg --with-keygrip --list-secret-keys
/home/grahamc/.gnupg/pubring.gpg
--------------------------------
sec rsa4096/0xFE918C3A98C1030F 2014-01-04 [SC] [expires: 2021-01-04]
BA94FDF11DA405212864C121FE918C3A98C1030F
Keygrip = 620D23E03A7B1FFD95BA9F14A36F861C759B61C1
uid [ultimate] Graham Christensen <[email protected]>
ssb rsa4096/0x8ED3C0087C86E062 2014-01-04 [E] [expires: 2021-01-04]
Keygrip = EAC1248DF7C760480853966B11DF4CF556AAA988
you can learn something (??? not sure what exactly) with gpg-connect-agent's janky protocol:
[grahamc@Petunia:~]$ gpg-connect-agent
> KEYINFO --ssh-list --ssh-fpr=sha256
S KEYINFO 3B493449D68C22E92DB9F9048E1E32A702364175 D - - 1 P SHA256:bgEmSPrLeuDOcM8QgjgzIfVlyRo1IeBTT9z1I8sAR8I - S
S KEYINFO DB1A6B6A24627F9D7551B21A9D51AB2C0376A4B2 D - - - C SHA256:9lJwrnDo0MI/osV7Eusz2kPWMVney09NFypkBFrzm28 - S
S KEYINFO A6FEF12F918389D277ACD84CF8B268835FA1D24A D - - - P SHA256:M2xGJvDL8hSCbe0GE6qDLBL7AOFPjq6C9iME6wRaZ58 - S
S KEYINFO B156709A4B3C39A6BC56E3DE74C194EEEA903CB6 D - - - P SHA256:TiIUAw/w6kByxDrWBXi5pszUrkMvXWwtGX5TVkzELSk - S
S KEYINFO 78973E39B5295662BAE63D3A3F8282B2BFABE4DA D - - - P SHA256:nsXG8wmjLBU6KlGob8vw9nwKgyb/kSWZAZbGVRnQCQM - S
S KEYINFO 52CF5EF6C4C5BAFC648D92F2DBF724FA887F6811 D - - - P SHA256:YMXioiuSN9I3EWMPAzZ4oHtgQCiygCOndHOs3Kn5xdk - S
OK
> KEYINFO --ssh-list --ssh-fpr
S KEYINFO 3B493449D68C22E92DB9F9048E1E32A702364175 D - - 1 P MD5:96:1d:d7:e3:28:5d:e6:9b:c0:34:b0:1a:01:4c:c9:a9 - S
S KEYINFO DB1A6B6A24627F9D7551B21A9D51AB2C0376A4B2 D - - - C MD5:f8:72:96:e7:ef:e4:a3:46:39:d4:86:bb:8e:1b:1b:4b - S
S KEYINFO A6FEF12F918389D277ACD84CF8B268835FA1D24A D - - - P MD5:3d:6d:a9:ac:76:e7:48:ad:4a:d0:50:81:46:2f:25:94 - S
S KEYINFO B156709A4B3C39A6BC56E3DE74C194EEEA903CB6 D - - - P MD5:ee:f1:24:85:d1:e2:a5:29:35:82:72:d0:02:f3:e3:14 - S
S KEYINFO 78973E39B5295662BAE63D3A3F8282B2BFABE4DA D - - - P MD5:6d:70:cd:3c:ed:4e:1e:d7:30:d9:44:6d:5c:e0:2e:92 - S
S KEYINFO 52CF5EF6C4C5BAFC648D92F2DBF724FA887F6811 D - - - P MD5:14:bd:7a:00:96:a4:14:8e:e3:7b:c6:7f:0d:29:c0:ff - S
OK
> %
and if you wanted to delete one, ssh-add won't work because gpg is bad, but:
$ gpg-connect-agent
DELETE_KEY one-of-those-long-strings-after-KEYINFO